Apparatus and method

ABSTRACT

A management unit including: a demand plan generating unit that obtains control information items related to energy consuming appliances; an obtaining unit that obtains picture data items showing the appliances; a memory which stores the picture data items obtained by the obtaining unit; a selecting unit which selects a picture data item from selection candidates including the picture data items stored in the memory, based on a specified control mode; a processor that executes processing for superimposing the control information items corresponding to the appliances in the picture data item selected by the selecting unit; and an output unit which displays the picture data item with the superimposed control information item.

TECHNICAL FIELD

The present invention relates to a control information displayapparatus, a control information display method, and a controlinformation display system.

BACKGROUND ART

There have been proposed systems in which a controller collects statesof energy use by appliances by communicating with the appliances, andprovides a user with degrees of appropriateness of the energy use by theappliances (for example, see Patent Literature 1).

CITATION LIST Patent Literature [PTL 1]

-   Japanese Patent Publication No. 3888906

[PTL 2]

-   PCT International Publication No. 2011/052696

SUMMARY OF INVENTION Technical Problem

However, when a control information item for satisfying a controlrequest in a building is presented, a user cannot specifically find outa target one of the appliances and the kind of control which should beperformed to achieve the presented control request.

In view of this, the present invention provides a control informationdisplay apparatus which presents a user with control informationnecessary for achieving a presented control request.

Solution to Problem

In order to solve the aforementioned conventional problem, aninformation display apparatus according to an aspect of the presentinvention includes: a first obtaining unit configured to obtain one ormore control information items regarding one or more appliances; asecond obtaining unit configured to obtain one or more picture dataitems showing images of the one or more appliances; a memory that storesthe one or more picture data items obtained by the second obtainingunit; a selecting unit configured to select, based on a specifiedcontrol mode, a picture data item from among selection candidatesincluding the one or more picture data items stored in the memory; aprocessor that executes processing for superimposing a controlinformation item for each of the one or more appliances which areincluded in the picture data item selected by the selecting unit, at aposition around the appliance in the picture data item; and a displayunit configured to display the picture data item with the controlinformation item superimposed thereon.

These general and specific aspects may be implemented using a system, amethod, an integrated circuit, a computer program, a computer-readablerecording medium such as a CD-ROM, or any combination of systems,methods, integrated circuits, computer programs, or recording media.

Advantageous Effects of Invention

The control information display apparatus according to the presentinvention is capable of presenting the user with the control informationnecessary for achieving the presented control request.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a configuration of an energy managementsystem in Embodiment 1.

FIG. 2 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 1.

FIG. 3 is a block diagram of a smart information superimposing unit inthe energy management system in Embodiment 1.

FIG. 4 is a flowchart for explaining a presentation need determiningprocess in Embodiment 1.

FIG. 5 is a diagram showing an example of a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 1.

FIG. 6 is a flowchart for explaining a presentation need determiningprocess in Embodiment 2.

FIG. 7 is a diagram showing an example of a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 2.

FIG. 8 is a diagram showing a configuration of an energy managementsystem in Embodiment 3.

FIG. 9 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 3.

FIG. 10 is a flowchart for explaining a presentation need determiningprocess in Embodiment 3.

FIG. 11 is a diagram showing an example of a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 3.

FIG. 12A is a flowchart for explaining a presentation need determiningprocess in Embodiment 4.

FIG. 12B is a second flowchart for explaining a presentation needdetermining process in Embodiment 4.

FIG. 12C is a third flowchart for explaining a presentation needdetermining process in Embodiment 4.

FIG. 13 is a diagram showing an example of a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 4.

FIG. 14 is a diagram showing a configuration of an energy managementsystem in Embodiment 5.

FIG. 15 is a block diagram of a smart information superimposing unit inthe energy management system in Embodiment 5.

FIG. 16 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 5.

FIG. 17A shows an example of a picture data item with superimposedpresentation data items output onto an output unit in Embodiment 5,specifically a picture data item showing a living room.

FIG. 17B shows an example of a picture data item with superimposedpresentation data items output onto the output unit in Embodiment 5,specifically a picture data item showing a bedroom 1 displayed after aswitch.

FIG. 17C shows an example of a picture data item with superimposedpresentation data items output onto the output unit in Embodiment 5,specifically a picture data item showing a bedroom 2 displayed after aswitch.

FIG. 18 is a diagram showing a configuration of an energy managementsystem in Embodiment 6.

FIG. 19 is a block diagram of a smart information superimposing unit inthe energy management system in Embodiment 6.

FIG. 20 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 6.

FIG. 21A is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 6, and in which a demand plan of “Plan 1” is selected.

FIG. 21B is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 6, and in which a demand plan of “Plan 2” is selected.

FIG. 21C is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 6, and in which a demand plan of “Plan 3” is selected.

FIG. 22 is a diagram showing a configuration of an energy managementsystem in Embodiment 7.

FIG. 23 is a block diagram of a smart information superimposing unit inthe energy management system in Embodiment 7.

FIG. 24 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 7.

FIG. 25A is a diagram of an example showing a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 7, in which a demand plan of “Plan 1” is selected.

FIG. 25B is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 7, and in which a demand plan of “Plan 2” is selected.

FIG. 25C is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 7, and in which a demand plan of “Plan 3” is selected.

FIG. 26 is a flowchart for explaining a presentation need determiningprocess in the case of prioritizing presentation of presentation dataitems related to energy consuming appliances whose states are to beadjusted in Embodiment 7.

FIG. 27 is a diagram showing a configuration of an energy managementsystem in Embodiment 8.

FIG. 28 is a block diagram of an appliance content unit in the energymanagement system in Embodiment 8.

FIG. 29 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 8.

FIG. 30A is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on an output unit inEmbodiment 8, and in which an appliance content is applied to one (airconditioner) of the energy consuming appliances.

FIG. 30B is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on the output unit inEmbodiment 8, and in which an appliance content is applied to one(television receiver) of the energy consuming appliances.

FIG. 30C is a diagram of an example which shows a picture data item withsuperimposed presentation data items output on the output unit inEmbodiment 8, and in which an appliance content is applied to one(lighting equipment) of the energy consuming appliances.

FIG. 31 is a diagram showing a configuration of an energy managementsystem in Embodiment 9.

FIG. 32 is a block diagram of an appliance content unit in the energymanagement system in Embodiment 9.

FIG. 33 is a sequence diagram for explaining operations in the energymanagement system in Embodiment 9.

FIG. 34A is a schematic diagram of an example which shows a picture dataitem with superimposed presentation data items output on an output unitin Embodiment 9, and in which the picture data item is output on theoutput unit in synchronization with an operation of one (airconditioner) of the energy consuming appliances.

FIG. 34B is a schematic diagram of an example which shows a picture dataitem with superimposed presentation data items output on the output unitin Embodiment 9, and in which one (air conditioner) of the energyconsuming appliances is performing a predetermined operation insynchronization with the output on the output unit.

FIG. 35 is a diagram of a management unit.

DESCRIPTION OF EMBODIMENTS (Underlying Knowledge Forming Basis of thePresent Disclosure)

The Inventors found that the energy-related information presentingsystem identified in the “Background Art” section has problems below.

Energy saving efforts have been made globally in order to reduce CO2 andprevent global warming. As examples of such efforts, nations are makingpromotion movements for saving energy by presenting specific numericaltargets to enterprises and homes.

In addition, due to reduction in electric power supply in Japan,enterprises and households have been requested to reduce electric powerconsumption drastically. If such reduction is not achieved, alarge-scale blackout may occur due to shortage of power supply withrespect to demands. This has been a social problem.

In view of this, there have been systems below. For example, a systemhas been proposed which prompts a user to save energy by measuringenergy consumption in his or her home and displaying the measurements ona monitor or the like. In the proposed system, a controller communicateswith appliances in the home and performs control for preventingunnecessary use of an appliance included in the appliances.

However, when a 15% reduction in power consumption in a household ispresented as a target, the household cannot find out how to achieve the15% reduction. Methods for using individual appliances in an energysaving manner have been introduced through various kinds of media.However, different households use different appliance configurations.For this reason, each of the households cannot find out how to combinesome of the methods for the appliance configuration in order to achievethe 15% reduction. For this reason, there are cases where the 15%reduction is not achieved actually, energy saving is performed too much,comfort is significantly affected, health is affected, etc.

In addition, systems have been proposed which includes the system (seePatent Literature 1) for collecting states of energy consumed by theappliances by communicating with the appliances and presenting the userdegrees of appropriateness of the energy use by the respectiveappliances. However, the household have many appliances without anycommunication function. These appliances are excluded from such targetsfor such presentation of the degrees of appropriateness of the energyuse. Thus, the user cannot find out how to achieve the 15% reduction.Supposing that the user can obtain a list of methods for using theappliances so as to achieve the 15% reduction in the household, the useris required to take complicated operations of searching out, one by one,the actual appliances corresponding to the respective appliances on thelist and using the appliances according to the specified methods.

In other words, when the control information item is presented tosatisfy the control request in the building, the user cannotspecifically find out a target one of the appliances and the kind ofcontrol which should be performed to achieve the presented controlrequest.

In view of this, the present invention provides a control informationdisplay apparatus which presents a user with control informationnecessary for achieving a presented control request.

In order to solve such a problem, an information display apparatusaccording to an aspect of the present invention is an informationdisplay apparatus including: a first obtaining unit configured to obtainone or more control information items regarding one or more appliances;a second obtaining unit configured to obtain one or more picture dataitems showing images of the one or more appliances; a memory that storesthe one or more picture data items obtained by the second obtainingunit; a selecting unit configured to select, based on a specifiedcontrol mode, a picture data item from among selection candidatesincluding the one or more picture data items stored in the memory; aprocessor that executes processing for superimposing a controlinformation item for each of the one or more appliances which areincluded in the picture data item selected by the selecting unit, at aposition around the appliance in the picture data item; and a displayunit configured to display the picture data item with the controlinformation item superimposed thereon.

In order to solve such a problem, a control information displayapparatus according to an aspect of the present invention is a controlinformation display apparatus (a management unit 100) including: a firstobtaining unit (a demand plan generating unit 106) configured to obtainone or more control information items (i) regarding one or moreappliances which consume energy provided in a building and (ii) used forsatisfying a control request in the building; a second obtaining unit (acamera unit 102, an appliance identification unit 104) configured toobtain one or more picture data items showing images of the one or moreappliances; a memory that stores the one or more picture data itemsobtained by the second obtaining unit; a selecting unit configured toselect, based on a specified control mode, a picture data item fromamong selection candidates including the one or more picture data itemsstored in the memory; a processor (a presentation data generating unit302, a superimposing unit 303) that generates a presentation data itemto be presented to a user to indicate a control information item foreach of the one or more appliances which are included in the picturedata item selected by the selecting unit, and superimpose thepresentation data item at a position around the appliance in the picturedata item; and a display unit configured to display the picture dataitem with the presentation data item superimposed thereon. It is to benoted that the control information display apparatus may include anobtaining unit 102 x including the camera unit 102 and the applianceidentification unit 104. In addition, the control information displayapparatus may include an image processing unit 107 x including thepresentation data generating unit 302 and the superimposing unit 303(FIG. 35 ).

For example, the second obtaining unit may be configured to obtain thepicture data items which include, as attribute information items of thepicture data items, image capturing position information itemsindicating image capturing positions of the picture data items, and theselecting unit may be configured to select, as the picture data item, aselection candidate associated with an image capturing position closerto a position of the control information display apparatus, the imagecapturing position being included in the image capturing positionsindicated by the image capturing information items.

For example, the control information display apparatus may furtherinclude: a control unit configured to control the one or more appliancesvia a communication network; an input unit configured to receive acontrol instruction for controlling a control target appliance among theone or more appliances; and a remote control application which transmitsthe control instruction received by the input unit to the control unit,wherein the control unit may be configured to control the currentcontrol target appliance based on the control instruction received fromthe remote control application.

For example, the a processor may be configured to superimpose, at aposition around the control target appliance in the picture data item,an activation button for activating the remote control applicationcorresponding to the current control target appliance which is includedin one of the picture data items, the remote control application maytransmit, to the control unit, the control instruction received from theinput unit, in response to a press of the activation button foractivating the remote control application, and the control unit may beconfigured to control the current control target appliance.

For example, the second obtaining unit may further be configured toobtain one or more video data items in each of which images of the oneor more appliances are included, and the selecting unit may beconfigured to select the picture data item from among the selectioncandidates which include (i) the video data items and (ii) the picturedata items stored in the memory.

For example, the control information display apparatus may furtherinclude: a third obtaining unit configured to obtain current valuesindicating amounts of energy consumed by the one or more appliances; anda presentation need determining unit configured to determine one or moreappliances which are included in the one or more appliances in thepicture data item, to be M or less appliances for each of which apresentation data item is to be presented to the user, M being apredetermined number (which is an integer), wherein the presentationneed determining unit may be configured to preferentially determine theM or less appliances included in the one or more appliances in thepicture data item, in descending order of differences between thecurrent values indicating the amounts of consumed energy and estimatedvalues indicating amounts of energy consumable in the case of performingcontrol based on the control information item, and the processor may beconfigured to superimpose the presentation data item onto the picturedata item, for each of the M or less appliances for which thepresentation data item is to be presented to the user.

For example, the control information display apparatus may furtherinclude a control unit configured to control one or more appliancescontrollable via a communication network among the appliances in thepicture data item, based on the control information items obtained bythe first obtaining unit, wherein the presentation need determining unitmay be configured to preferentially determine, as the M or lessappliances, one or more appliances uncontrollable via the communicationnetwork over the one or more appliances controllable via thecommunication network among the appliances in the picture data item.

For example, the processor may be configured to generate, as thepresentation data items, a detailed presentation data item includingdetailed information, and a brief presentation data item including briefinformation, and when determining the M or less appliances, thepresentation need determining unit may be configured to: preferentiallydetermine one or more appliances included in the one or more appliancesin the picture data item, to be Md or less appliances for each of whichthe detailed presentation data item is to be presented, in descendingorder of differences between the current values indicating the amountsof consumed energy and estimated values indicating amounts of energyconsumable in the case of performing control based on the controlinformation item, Md being a predetermined number (which is an integersatisfying Md M); and preferentially determine one or more appliancesother than the Md or less appliances for each of which the detailedpresentation data item is to be presented among the one or moreappliances in the picture data item, to be (M−Md) or less appliances foreach of which the brief presentation data item is to be presented, indescending order of differences between the current values indicatingthe amounts of consumed energy and estimated values indicating amountsof energy, (M−Md) denoting a predetermined number.

For example, the first obtaining unit may be configured to receive thecontrol request from an electric power company or a public institutionvia a network.

For example, the display unit may include a display screen, and thepresentation need determining unit is configured to determine thepredetermined number M according to a size of the display screen.

For example, an operation unit may be configured to receive aninstruction from the user, wherein the presentation need determiningunit may be configured to determine the number received through theoperation unit as the predetermined number M.

For example, the control information display apparatus is one of amobile phone, a smart phone, and a television receiver.

For example, the first obtaining unit may be configured to obtain aplurality of the control information items regarding a plurality of theappliances, the second obtaining unit may be configured to obtain, asthe one or more picture data items, a plurality of the picture dataitems showing images of the appliances, and the processor may beconfigured to generate at least one of a plurality of the presentationdata items which is for presenting a corresponding one of the controlinformation items which is for at least one of the appliances, andsuperimpose the at least one presentation data item onto a correspondingone of the picture data items, in association with the at least oneappliance whose location is identified in the corresponding picture dataitem.

For example, the control information display apparatus may furtherinclude a presentation priority determining unit configured to set, foreach of the appliances included in the picture data item, a presentationpriority level to the at least one presentation data item for presentingthe corresponding control information item, the presentation prioritylevel becomes higher with an increase in a difference between theestimated value indicating the amount of energy consumable in the caseof performing the control based on the control information item and thecurrent value indicating the amount of consumed energy, wherein theprocessor may be configured to superimpose the at least one presentationdata item having the higher presentation priority level set by thepresentation priority determining unit, in a larger size onto thedisplay unit.

For example, a plurality of the control requests may include a controlrequest regarding a current amount of energy consumed in the building.

For example, a plurality of the control requests may include a controlrequest regarding an indoor environment parameter including at least atemperature, a humidity, or an luminance in the building.

For example, a plurality of the control requests may include a controlrequest regarding an indoor air quality in the building.

For example, the second obtaining unit may be configured to obtain, asthe picture data item, a picture data item with an identificationinformation item including ID and a position of the appliance in thepicture data item, and the a processor may be configured to superimposethe presentation data item at the position around the appliance in thepicture data item identified by the identification information item.

In addition, a control information display apparatus according to anaspect of the present invention includes a first obtaining unitconfigured to obtain a control request regarding amounts of energyconsumed in a building; a second obtaining unit configured to obtain oneor more picture data items showing images of one or more applianceswhich consume energy provided in the building, each of the one or morepicture data items including identification information items of the oneor more appliances; a third obtaining unit configured to obtain currentvalues indicating amounts of energy consumed by the one or moreappliances; a control information generating unit configured to generatea control information item for each of the one or more appliances tosatisfy the control request, based on the current values indicating theamounts of energy consumed by the one or more appliances; a processorthat generates at least one presentation data item for presenting thecontrol information item of at least one of the one or more applianceswhose images are included in the picture data item, and superimpose theat least one presentation data item onto the image data item inassociation with the at least one appliance identified by theidentification information item; and a display unit configured todisplay the picture data item with the at least one presentation dataitem superimposed thereon.

In addition, a control information display method according to an aspectof the present invention includes: obtaining one or more controlinformation items (i) regarding one or more appliances which consumeenergy provided in a building and (ii) used for satisfying a controlrequest in the building; obtaining one or more picture data itemsshowing images of the one or more appliances; storing the one or morepicture data items obtained in the obtaining of the one or more picturedata items; selecting, based on a specified control mode, a picture dataitem from among selection candidates including the one or more picturedata items stored in the storing; generating a presentation data item tobe presented to a user to indicate a control information item for eachof the one or more appliances which are included in the picture dataitem selected in the selecting, and superimposing the presentation dataitem at a position around the appliance in the picture data item; anddisplaying the picture data item with the presentation data itemsuperimposed thereon.

In addition, a control information display system according to an aspectof the present invention includes appliances which consume energyprovided in a building, the system including a control informationdisplay apparatus including: a first obtaining unit configured to obtainone or more control information items (i) regarding one or moreappliances which consume energy provided in a building and (ii) used forsatisfying a control request in the building; a second obtaining unitconfigured to obtain one or more picture data items showing images ofthe one or more appliances; a memory that stores the one or more picturedata items obtained by the second obtaining unit; a selecting unitconfigured to select, based on a specified control mode, a picture dataitem from among selection candidates including the one or more picturedata items stored in the memory; a processor that generates apresentation data item to be presented to a user to indicate a controlinformation item for each of the one or more appliances which areincluded in the picture data item selected by the selecting unit, andsuperimpose the presentation data item at a position around theappliance in the picture data item; and a display unit configured todisplay the picture data item with the presentation data itemsuperimposed thereon.

With the control information display apparatus, the presentation dataitems based on a demand plan is superimposed around the target energyconsuming appliance on the output unit. Thus, the user can easilyunderstand how to use the energy consuming appliance so as to satisfy ademand control request.

When many energy consuming appliances are displayed on the displayscreen, a limited number of presentation data items are superimposedonto the picture data item so as to sufficiently secure the visibility.For example, presentation data items related to energy consumingappliances having larger differences between predetermined operationstates and current operation states are preferentially superimposedbased on the demand plan because the presentation data items providehigher energy saving effects when displayed.

In this case, the presentation data items superimposed on the data onthe output unit are visually recognized by the user. Thus, effectivecontrol based on the presentation data items is performed to achieve thedemand control request. Accordingly, it is possible to prompt the userto use the energy consuming appliances effectively to achieve the demandcontrol request.

In addition, it is possible to provide an energy control apparatus whichprovides control information items related to the appliances in a highvisible manner in order to achieve the energy consumption target in thebuilding.

Embodiment 1

FIG. 1 is a diagram showing a configuration of an energy managementsystem in this embodiment.

The energy management system in this embodiment includes n energyconsuming appliances 151 to 15 n (n is an integer larger than or equalto 1) and a management unit 100.

Each of the energy consuming appliances 151 to 15 n is placed in a home,and provides particular functions for the user while consuming energy.The management unit 100 manages energy consumed by the energy consumingappliances 151 to 15 n.

The management unit 100 includes: an appliance observing unit 101; acamera unit 102; an appliance information storing unit 103; an applianceidentifying unit 104; an input unit 105; a demand plan generating unit106; a smart information superimposing unit 107; and an output unit 108.

The appliance observing unit 101 communicates with the energy consumingappliances wired or wirelessly, obtains and stores the amounts of energyconsumed by the energy consuming appliances and appliance operationstate information items related to the states of the respective energyconsuming appliances.

The camera unit 102 captures images of the energy consuming appliancesplaced inside and outside of the home.

The appliance information storing unit 103 stores (i) appliancespecifications information items including the manufactures, models,power consumptions etc. of the energy consuming appliances 151 to 15 nplaced in the home, and (ii) appliance identification information itemsfor identifying the respective energy consuming appliances.

The appliance identifying unit 104 identifies the energy consumingappliance (specifically, the ID of the energy consuming appliance andthe image position of the energy consuming appliance) included in thepicture data item obtained by the camera unit 102. The applianceidentifying unit 104 may obtain a picture data item from an externalappliance (not shown) and identify the energy consuming applianceincluded in the picture data item. In this case, the camera unit 102 isunnecessary.

The input unit 105 receives an input of the demand control requestrelated to energy consumed in the home.

The demand plan generating unit 106 generates the demand plan of detailsof control for the respective energy consuming appliances 151 to 15 n inorder to satisfy the demand control request received through the inputunit 105. The demand plan generating unit 106 generates informationitems to be presented to the user (hereinafter referred to as“presentation information items”) including the demand plan (see FIG. 3which is described later).

The smart information superimposing unit 107 superimposes presentationdata items generated from the presentation information items onto apicture data item showing energy consuming appliances identifiedtherein.

The output unit 108 outputs the picture data item with presentation dataitems superimposed by the smart information superimposing unit 107.

The energy consuming appliances 151 to 15 n are home appliances such asan air conditioner, a lighting equipment, and a television receiver.These home appliances may be a hot water dispenser, a solar powergenerating appliance, power storage appliance, or so on. The solar powergenerating appliance is included in the energy consuming appliances andgenerates electric energy and whose energy consumption is negative.

The camera unit 102 is a digital camera capable of imaging a still imageand a video.

The appliance observing unit 101, the appliance information storing unit103, the appliance identifying unit 104, the input unit 105, the demandplan generating unit 106, and the smart information superimposing unit107, and the output unit 108 are configured actually in the forms of oneor more CPUs and a plurality of memory appliances. In other words, theone or more CPUs execute the program stored in the memory appliances,and stores information on memory areas on the memory appliances inreadable formats. This makes it possible to realize the functions of theappliance observing unit 101, the appliance information storing unit103, the appliance identifying unit 104, the demand plan generating unit106, and the smart information superimposing unit 107.

In addition, the input unit 105 includes, as elements, a touch panel andan audio microphone. Each CPU executes driver software stored in thememory appliances, a soft keyboard, and a speech recognitionapplication. In this way, the one or more CPUs receive various kinds oftouch inputs from the touch panel and audio inputs from the audiomicrophone.

In addition, the output unit 108 includes, as elements, a liquid crystaldisplay and a speaker. Each CPU executes the driver software stored inthe memory appliances, a display application such as a browser, and anaudio reproduction application. In this way, the one or more CPUsdisplay information onto the liquid crystal display and output audiofrom the speaker.

Here, the touch panel of the input unit 105 and the liquid crystaldisplay of the output unit 108 constitute the same display screen. Forexample, the input unit 105 receives an input for information output onthe output unit 108. More specifically, for example, when an operationsuch as a tap operation is performed on the touch panel of the inputunit 105, the operation is received as an input by an applianceapplication such as a browser executed by the CPU. The operation isreceived as an operation of pressing a button displayed at a positionsuch as a tap position, or as an operation of selecting a link displayedon the position.

The appliance observing unit 101, the camera unit 102, the applianceinformation storing unit 103, the appliance identifying unit 104, theinput unit 105, the demand plan generating unit 106, the smartinformation superimposing unit 107, and the output unit 108 areconfigured in the form(s) of one or more appliances. When these unitsare configured as a plurality of appliances, these appliances areconnected to each other for wired or wireless communication.

FIG. 2 is a diagram showing a sequence of operations performed by theenergy management system in this embodiment.

First, the appliance observing unit 101 executes an appliance observingprocess 201. In the appliance observing process 201, the applianceobserving unit 101 obtains, through communication with each of theenergy consuming appliances 151 to 15 n, the amounts of energy consumedby the respective energy consuming appliances and information itemsrelated to the states of the respective energy consuming appliances(hereinafter referred to as “appliance operation state informationitems”), and stores the amounts and the information items on some of thememory areas. The appliance observing unit 101 executes an applianceobserving process 201 repeatedly and periodically.

Meanwhile, the user inputs the demand control request 203 through theinput unit 105 (an input process 202). For example, the demand controlrequest 203 is a demand request related to energy consumed in the home.The case where the demand control request 203 is a demand controlrequest related to energy consumed in the home is described below.

The demand control request 203 which has been input is transmitted tothe demand plan generating unit 106. Here, the demand control request203 is a request for “a 10% reduction in the energy consumption amountin the entire home”, “a 10% reduction in the CO2 discharge amount due toenergy consumption in the home”, or “a 10% reduction in cost requiredfor energy consumption in the home”. Hereinafter, for simplicity,descriptions are given of a case where the demand control request 203 isthe request for “a 10% reduction in the energy consumption amount in theentire home”.

Upon receiving the demand control request 203, the demand plangenerating unit 106 performs a demand plan generating process 206. Inthe demand plan generating process 206, access to the applianceobserving unit 101 is made, and the appliance operation stateinformation items 204 stored in one of the memory areas are referred to.Furthermore, the appliance specifications information items 205 storedin the appliance information storing unit 103 are referred to. Withreference to these information items, the demand plan satisfying thedemand control request 203 is generated using a method such as apublicly known method.

The appliance information storing unit 103 stores the appliancespecifications information items 205 of the respective energy consumingappliances 151 to 15 n. More specifically, information items related toenergy consumed in various kinds of operation states of the energyconsuming appliances are stored as the appliance specificationsinformation items 205 of the respective energy consuming appliances. Inthis storing operation, for example, the appliance specificationsinformation items 205 are stored in association with ID of the energyconsuming appliance.

Examples of publicly known methods that the demand plan generating unit106 can use to generate the demand plan include the method as indicatedbelow. Specifically, the method is a method in which a simulationexecuting unit generates a reduction plan as described in the paragraphs[0078] and [0106] of Patent Literature 2. In the case of the demand plangenerating unit 106 in this embodiment, the following simulation methodis shown as an example. For example, there are control patterns whichcan be combined with a current configuration of energy consumingappliances. In other words, the current configuration of energyconsuming appliances is combined with each of the control patterns. Thedemand plan generating unit 106 performs a simulation for each of thecombinations. The demand plan generating unit 106 evaluates the resultsof the simulations, derives one of the control patterns which yields aresult satisfying the demand control request 203 (corresponding to thereduction target in Patent Literature 2), and determines details of thederived control pattern as the demand plan.

The demand plan generating unit 106 transmits, to the smart informationsuperimposing unit 107, the presentation information items 207 includingthe generated demand plan and the appliance operation state informationitems 204 indicating current operation states of the respective energyconsuming appliances.

Next, when the user captures images of the interior or exterior of thehome using the camera unit 102 (an image capturing process 211), thepicture data item 212 is transmitted to the appliance identifying unit104.

The appliance identifying unit 104 which has received the picture dataitem 212 performs an appliance identifying process 214. In this process,the appliance identifying unit 104 identifies the energy consumingappliance included in the received picture data item 212, with referenceto the (earlier-mentioned) appliance identification information items213 stored in the appliance information storing unit 103.

The appliance information storing unit 103 stores the applianceidentification information items 213 of the respective energy consumingappliances 151 to 15 n. The appliance identification information item213 of each of the energy consuming appliance is an information item foridentifying the energy consuming appliance through image analysis fromamong the picture data item 212 including the energy consumingappliance. This information item is stored in association with the ID ofthe energy consuming appliance.

The appliance identifying unit 104 performs image processing on thepicture data item 212 to extract feature information. The applianceidentifying unit 104 verifies whether the feature information matchesthe appliance identification information items 213 stored in theappliance information storing unit 103. In this way, the applianceidentifying unit 104 identifies the energy consuming appliance includedin the picture data item 212 to be the energy consuming appliancecorresponding to the appliance identification information item whoseverification has been successfully done. The appliance identifying unit104 transmits, to the smart information superimposing unit 107, thepicture data item 215 with the identification information of theidentified energy consuming appliance obtained by adding theidentification information of the appliance to the picture data item 212obtained by the camera unit 102. The appliance identificationinformation includes ID of the identified energy consuming appliance andthe information indicating a position of the energy consuming appliancein the picture data item 212 in a video.

Next, upon receiving the presentation information items 207 and thepicture data item 215 with the identification information items, thesmart information superimposing unit 107 executes a smart superimposingprocess 216. In this process, presentation data items which are to beactually presented to the user are generated from the presentationinformation items 207 based on a predetermined algorithm which is laterdescribed. The generated presentation data items are superimposed aroundthe (afore-mentioned) positions of the energy consuming appliances inthe picture data item 215 with the identification information items. Atthis time, in the smart superimposing process 216, a limited number ofpresentation data items are to be superimposed considering uservisibility. In this way, the details of the presentation informationitems 207 are displayed in a comparatively easy-to-recognize format. Forexample, the picture data item 215 with the identification informationitems is transmitted from the appliance identifying unit 104 to thesmart information superimposing unit 107. As another example, thepicture data item 215 with the identification information items may betransmitted from an external appliance which functions as a camera unit102 and an appliance identifying unit 104 to the smart informationsuperimposing unit 107. Here, “around” means within a range having apredetermined distance from a specified position.

In addition, the smart superimposing process 216 performs a process forgenerating picture data item 217 on which the generating picture dataitem 215 with the identification information items and the presentationdata items have been superimposed. These data items are to be output bythe display application of the output unit 108, the audio reproductionapplication etc. More specifically, the smart superimposing process 216generates data items having a data format below. Specifically, the dataitems are input through the display application of the output unit 108and the audio reproduction application and have a data format which canbe interpreted by the display application and the audio reproductionapplication. These data items are included in the picture data item 217with superimposed presentation data items. More specifically, in thesmart superimposing process 216, the data items generated here are dataitems on which the picture data item 215 and the presentation data itemsare superimposed, and which are written using Hyper Text Markup Language(HTML) or the like. For example, the smart information superimposingunit 107 generates, as the picture data item 217 with superimposedpresentation data items, the data items written using HTML and dataitems such as image data items and audio data items referred to by theHTML data items, and outputs the generated data items to the output unit108.

Next, the output unit 108 outputs the picture data item 217 with thesuperimposed presentation data items using the liquid crystal displayand the speaker (an output process 218). Based on the HTML data items,the display application of the output unit 108 displays the presentationdata items on the liquid crystal display by superimposing them aroundthe energy consuming appliances. In addition, the audio reproductionapplication of the output unit 108 drives the speaker to reproduce theaudio data items referred to by the HTML data items.

Next, the smart information superimposing unit 107 is described indetail.

FIG. 3 shows an internal structure of the smart informationsuperimposing unit 107.

The smart information superimposing unit 107 includes: a presentationneed determining unit 301; a presentation data generating unit 302; anda superimposing unit 303. The presentation need determining unit 301determines whether each of presentation data items needs to be presentedto the user based on the predetermined algorithm (a presentation needdetermining process). The presentation data generating unit 302generates presentation data items determined to be presentation dataitems which need to be presented by the presentation need determiningunit 301, based on the presentation information item. The superimposingunit 303 superimposes the presentation data items determined to bepresentation data items which need to be presented by the presentationneed determining unit 301 onto the picture data item 215 with theappliance identification information items of the identified energyconsuming appliances.

For example, there is a possibility that a comparatively large number ofenergy consuming appliances is identified in the picture data item 212.In this case, when presentation data items for the respective energyconsuming appliances are simply superimposed, many presentation dataitems are displayed on the liquid crystal display at the same time. Thismakes it difficult for the user to recognize the data items, resultingin a reduction in visibility. In view of this, the presentation needdetermining unit 301 limits the number of presentation data items to besuperimposed, by determining whether each of the information itemsindicating the details of the demand plan needs to be presented to theuser, based on the predetermined algorithm with consideration of uservisibility.

FIG. 4 is a flowchart of the presentation need determining processperformed by the presentation need determining unit 301.

First, the presentation need determining unit 301 counts the number N (Nis an integer larger than or equal to 0) of energy consuming appliancesidentified in the picture data item 212, based on applianceidentification information items of the appliances (Step S401).

Next, the presentation need determining unit 301 quantifies thedifferences between target values obtainable by performing the demandplan and current values related to the identified N energy consumingappliances (Step S402). Examples of data items specifically used hereinclude control details for the energy consuming appliances indicated bythe demand plan. In the case where the demand control request is acontrol request related to a total energy consumption amount, thedifference ΔE (=Er−Ep) between an estimated value Ep and an actual valueEr is calculated as being obtainable value when the control details areperformed, wherein Ep denotes the amount of energy consumed by eachenergy consuming appliance and Er denotes the amount of energy which isbeing consumed in a current state.

In the case where the demand control request is a control requestrelated to a CO2 discharge amount, the difference ΔE (=Er−Ep) between anestimated value Ep and an actual value Er is calculated as beingobtainable value when the control details are performed, wherein Epdenotes a CO2 discharge amount based on energy consumed by each energyconsuming appliance and Er denotes a CO2 discharge amount based onenergy which is being consumed in a current state.

In the case where the demand control request is a control requestrelated to cost, the difference ΔE (=Er−Ep) between an estimated valueEp and an actual value Er is calculated as being obtainable value whenthe control details are performed, wherein Ep denotes cost based onenergy consumed by each energy consuming appliance and Er denotes costbased on energy which is being consumed in a current state.

Next, the number N of energy consuming appliances is compared with themaximum presentation number M (M is an integer larger than or equalto 1) indicating the number of presentation data items to be presentedat maximum (Step S403). Here, the maximum presentation number M is theupper limit value for the number of presentation data items to bepresented on the liquid crystal display of the output unit 108 at thesame time. In other words, the maximum presentation number M is a valuepredetermined based on the display screen size and resolution of theliquid crystal display, as a value for sufficiently securing uservisibility even when the presentation data items are displayed at thesame time.

When N>M is satisfied, the number of energy consuming appliancesidentified in the picture data item is larger than the maximumpresentation number M. For this reason, M energy consuming appliancesamong the identified appliances are selected in descending order of thedifferences ΔE between the values in the demand plan and the currentvalues in Step S402, and are determined to be the energy consumingappliances for which presentation data items need to be presented forcompletion of the presentation need determining process (Step S404).Here, ΔE is also referred to as a presentation priority level. Theprocess can be expressed as a process for selecting the M energyconsuming appliances selected in descending order of presentationpriority levels of the presentation data items and determining thepresentation data items related to the energy consuming appliances aspresentation data items which need to be presented.

When N≤M is satisfied, the number of energy consuming appliancesidentified in the picture data item is smaller than or equal to themaximum presentation number M. For this reason, presentation data itemsof N energy consuming appliances among the appliances identified in thepicture data item are determined to be presentation data items whichneed to be presented, for completion of the presentation needdetermining process (Step S405).

Accordingly, through the aforementioned presentation need determiningprocess, the following operations are performed.

For example, when a slight change in the orientation of the camera unit102 increases the number N of energy consuming appliances identified inthe picture data item 212, some of the N energy consuming applianceswhose presentation data items once determined to be presentation targetsmay be excluded from the presentation targets. As another example, whena slight change in the orientation of the camera unit 102 decreases thenumber N of energy consuming appliances identified in the picture dataitem 212, energy consuming appliances which have not been included inthe N energy consuming appliances whose presentation data items oncedetermined to be presentation data items which need to be presented maybe included in the presentation targets.

Furthermore, for example, when a zoom-out of the camera unit 102increases the number N of energy consuming appliances identified in thepicture data item 212, some of the N energy consuming appliances whosepresentation data items once determined to be presentation data itemswhich need to be presented may be excluded from the presentationtargets. As another example, when zoom-in of the camera unit 102decreases the number N of energy consuming appliances identified in thepicture data item 212, energy consuming appliances which have not beenincluded in the N energy consuming appliances whose presentation dataitems once determined to be presentation targets may be included in thepresentation targets.

Furthermore, for example, when a zoom-out of the camera unit 102increases the number N of energy consuming appliances identified in thepicture data item 212, some of the N energy consuming appliances whosepresentation data items once determined to be presentation targets maybe excluded from the presentation targets. In this case, thepresentation data items related to the energy consuming appliances arenot presented to the user. As another example, when zoom-in of thecamera unit 102 decreases the number N of energy consuming appliancesidentified in the picture data item, energy consuming appliances whichhave not included in the N energy consuming appliances whosepresentation data items once determined to be presentation targets maybe included in the presentation targets. In this case, the presentationdata items related to the energy consuming appliances are presented tothe user.

In the presentation need determining process, it is also good todetermine the needs for presenting the presentation data items of onlyenergy consuming appliances having a particular function such as anenergy saving mode or energy consuming appliances made by a particularmanufacturer.

In this case, the appliance identifying unit 104 performs a process ofadding appliance identification information items to picture data itemsof the energy consuming appliances having a particular function such asan energy saving mode or the energy consuming appliances made by theparticular manufacturer, with reference to specifications informationitems stored in the appliance information storing unit 103.

In addition, the presentation data generating unit 302 generatespresentation data items 302 a determined to be presented by thepresentation need determining unit 301. The generated presentation dataitems 302 a includes at least one of picture data items, audio dataitems, text data items, video data items, etc.

When a rate ((Er−Ep)/Ep) of the difference between Er and Ep withrespect to the demand plan calculated in Step S402 is larger than orequal to a predetermined rate, the presentation data generating unit 302performs operations below. Specifically, the presentation datagenerating unit 302 generates, as the presentation data items 302 a,information items indicating current operation states, and a piece ofadvice for prompting control details indicated by the demand plan.

For example, when a rate ((Er−Ep)/Ep) of the difference between Er andEp with respect to the demand plan is larger than or equal to 10%, thepresentation data generating unit 302 generates, as the presentationdata item 302 a, an information item indicating a current operationstate, and a piece of advice for prompting control details indicated bythe demand plan.

In addition, when a rate ((Er−Ep)/Ep) of the difference between Er andEp with respect to the demand plan calculated in Step S402 is smallerthan the predetermined rate, the presentation data generating unit 302generates, as the presentation data item 302 a, an information itemindicating a current operation state, and an information item indicatingthat the current operation state is almost ideal for achieving thedemand control request.

For example, when the predetermined rate is 10% and a rate ((Er−Ep)/Ep)of the difference between Er and Ep with respect to the demand plan issmaller than 10%, the presentation data generating unit 302 generates,as the presentation data item 302 a, an information item indicating acurrent operation state, and an information item indicating that thecurrent operation state is almost ideal for achieving the demand controlrequest.

The presentation data generating unit 302 stores, in advance, textmessage templates, image data items, audio data items, and animationdata items which are component data items of the presentation data items302 a. The presentation data generating unit 302 generates thepresentation data items 302 a by embedding setting values for the demandplan included in the presentation information items 207, and combiningwith images of the component data.

The component data items include icons showing states of energyconsuming appliances, image data items or animation data items ofanthropomorphic characters, etc, and are used to present to the user thestates of the energy consuming appliances and a piece of adviceregarding the usage of the appliances in a comparativelyeasy-to-understand format.

The use of the comparatively easy-to-understand format makes it possibleto draw the user's attention to the presentation details effectively,increasing the possibility that the user actually performs the operationbased on the piece of advice of the presentation data item 302 a.

The superimposing unit 303 superimposes the presentation data items 302a generated by the presentation data generating unit 302 on the picturedata item 215 with appliance identification information items input fromthe appliance identifying unit 104.

At this time, the presentation data items 302 a corresponding to anidentified energy consuming appliance is superimposed around theidentified energy consuming appliance in the picture data item 215 withthe appliance identification information items.

At this time, the superimposing unit 303 generates the picture data item217 with the superimposed presentation data items. When this data isused through the display application and the audio reproductionapplication on the output unit 108, the picture data item 215 with thesuperimposed presentation data items 302 a is output. In other words,the data has a data format of Hyper Text Markup Language (HTML) or thelike which can be interpreted by the display application.

Next, FIG. 5 shows an example where the picture data item 217 with thesuperimposed presentation data items 302 a is output to the output unit108.

FIG. 5 is a diagram showing an example where a liquid crystal display511 and a speaker 512 of the mobile terminal 500, and the displayapplication (not shown) and the audio reproduction application (notshown) executed on the CPU of the mobile terminal 500 are configured asthe output unit 108. In addition, FIG. 5 shows an example where themaximum presentation number M for the presentation data items is 1, andthe presentation data item of the energy consuming appliance (airconditioner) 152 is determined to be the presentation data item whichneeds to be presented.

In this case, the picture data item 217 with the superimposedpresentation data items is displayed on the liquid crystal display 511by the display application. The picture data item 217 with thesuperimposed presentation data items is of a captured image of a homespace 500R in which the following are included: an energy consumingappliance (television receiver) 151; an energy consuming appliance (airconditioner) 152; and an energy consuming appliance (lighting equipment)153.

Furthermore, data items such as an image data item 501 which visuallyenhances the energy consuming appliance (air conditioner) 152 and a textdata item 502 of “Please increase setting temperature to 28° C.!” aredisplayed as presentation data items for the energy consuming appliance(air conditioner) 152 on the liquid crystal display 511 by the displayapplication.

In addition, when the user taps an image 152 u representing the energyconsuming appliance (air conditioner) on the touch panel integrated withthe liquid crystal display 511, the audio reproduction applicationoutputs a speech of “Please increase setting temperature to 28° C.!”from the speaker 512.

In this way, the presentation data based on the demand plan is outputaround the corresponding energy consuming appliance on the output unit108. For this reason, the user can easily understand how to use each ofthe energy consuming appliances so as to satisfy the demand controlrequest 203.

When the number of energy consuming appliances to be presented on thedisplay screen is comparatively large, a limited number of presentationdata items are to be superimposed onto corresponding picture data itemsso as to secure sufficient visibility. More specifically, presentationdata items related to energy consuming appliances having largerdifferences between operation states based on the demand plan andcurrent operation states are preferentially superimposed based on thedemand plan because the presentation data items provide higher energysaving effects when displayed.

Thus, the user visually recognizes, on the output unit 108, the picturedata items with the superimposed presentation data items which arecomparatively effective for achieving the demand control request. Inthis way, it is expected that the control details of the presentationdata items are executed in a comparatively secured manner.

Accordingly, it is possible to prompt the user to perform thecomparatively effective methods for using the energy consumingappliances to achieve the demand control request.

Here, it is also good to include image data items and animation dataitems of a fairy which is virtually attached to one of the energyconsuming appliances, in the component data items stored in thepresentation data generating unit 302, and the presentation datagenerating unit 302 may generate presentation data items using thesecomponent data items.

In this case, the fairy of the energy consuming appliance gives the useran impression as if the fairy of the energy consuming appliance ispresenting pieces of advice about a current state of the energyconsuming appliance and a method of using the energy consumingappliance. In this way, it is possible to present the presentation dataitems in a user friendly manner.

Furthermore, it is also good to activate the same operation as executedwhen the energy consuming appliance (air conditioner) 152 is tapped onthe touch panel when the user calls a preset name of the fairy of theenergy consuming appliance.

In this case, information related to the preset name of the fairy of theenergy consuming appliance is stored in the appliance informationstoring unit 103 together with a specifications information item and anappliance identification information item. A speech recognitionapplication of the input unit 105 recognizes the call of the preset nameof the fairy of the energy consuming appliance, with reference to theinformation stored in the appliance information storing unit 103.

For example, in the case of an example shown in FIG. 5 , in response toa call of the name of the fairy of the energy consuming appliance (airconditioner) 152, the audio reproduction application outputs the speechof “Please increase setting temperature to 28° C.!” from the speaker512.

Presenting such a presentation data item in the comparativelyeasy-to-understand format makes it possible to draw the user's attentionto the presentation details effectively, increasing the possibility thatthe user actually performs the operation based on the piece of advice ofthe presentation data item.

In the above description, the appliance observing unit 101 obtains theappliance operation state information items through wired or wirelesscommunication with the energy consuming appliances. Alternatively, it isalso good to set an electric power measurement appliance to adistribution switchboard, measure the amounts of energy consumed by therespective energy consuming appliances, and obtain the amounts of energythrough wired or wireless communication with the electric powermeasurement appliance.

Furthermore, it is also good to obtain information items related tocurrent states of the respective energy consuming appliance by analyzingtransitions in the amounts of energy consumed by the respective energyconsuming appliances.

In this case, it is possible to configure the energy management systemin this embodiment even when the energy consuming appliances do not haveany communication function for communicating with the applianceobserving unit 101.

In the above description, the demand control request is input by theuser through the input unit 105. However, the energy management systemmay be configured to receive a demand control request externally from anelectric power company or a public institution through a secure network.

In this case, it is possible to externally prompt each of households toperform the demand control request, and for example, it is possible toperform demand control at a local area level.

In addition, the maximum presentation number M may be automatically setaccording to the size and resolution of the liquid crystal displayscreen of the output unit 108. In this case, even in the case of outputon another output unit including a liquid crystal display having ascreen size and a different resolution, the maximum presentation numberM is automatically set to a value according to the liquid crystaldisplay, and thus a user visibility is sufficiently secured even whenthe M presentation data items are displayed at the same time.

In addition, the maximum presentation number M may be set to a valueinput by the user through the input unit 105. Visibility in the casewhere a plurality of presentation data items are displayed at the sametime varies from person to person. Thus, allowing setting of the valueinput by the user makes it possible to set the maximum presentationnumber M to a value which secures visibility for each user.

In addition, the maximum presentation number M may be automatically setto a number according to the liquid crystal display of the output unit108 and display functions of the display application.

In this case, in the case of output on the output unit such as a mobilephone having a comparatively small display function, the maximumpresentation number M is automatically set to the number according tothe display function. Thus, even when presentation data items aredisplayed at the same time, display is performed without any extremedelay, and a user visibility is secured.

Here, it is also good to attach identification markers to the energyconsuming appliances 151 to 15 n, and stores, in the applianceinformation storing unit 103, the markers and the ID of the energyconsuming appliances as appliance identification information items forthe energy consuming appliances in a mutually associated manner.

In this case, it is possible to identify the appliances with highaccuracy.

Here, it is also good to provide a plurality of camera units 102 atdifferent positions in a home so that picture data items input to theappliance identifying unit 104 can be selected by switching from among aplurality of picture data items obtained by the camera units 102 at thedifferent positions in the home.

In this case, when the presentation need determining unit 301 determinesa presentation data item which needs to be presented in a superimposedmanner from the output unit 108, the presentation data item is outputaccordingly by making a switch to a corresponding one of the picturedata items 212 input, from the camera units 102 to the applianceidentifying unit 104, for the energy consuming appliances which are atthe different positions in the home and whose images are included at thedifferent positions in the picture data item.

Here, the mobile terminal 500 may be, for example, a mobile phone, asmart phone, a television receiver, a wrist-watch-type mobile terminal,or a glass-type mobile terminal. For example, when the mobile terminal500 is the glass-type mobile terminal, a presentation data item isdisplayed in a superimposed manner on a liquid crystal display of a lensportion.

Here, the presentation data generating unit 302 may generatepresentation formats for the presentation data items together with thepresentation data items 302 a. At this time, the presentation datagenerating unit 302 may set different presentation formats according topresentation priority levels of the presentation data items. Morespecifically, presentation formats may be generated which are designedto display an image data item, a text data item, or a video data itemwhich is included in the presentation data items 302 a and has a largerpresentation priority level is displayed in a comparatively large size.In addition, presentation formats may be generated which are designed tooutput, in a comparatively large size, a speech presented by an audiodata item which is included in the presentation data items 302 a and hasa larger presentation priority level. In other words, presentationformats may be generated which are designed to display, in acomparatively small size, an image data item, a text data item, or avideo data item which is included in the presentation data items 302 aand has a smaller presentation priority level. In addition, presentationformats may be generated which are designed to output a speech presentedby an audio data item which is included in the presentation data items302 a and has a smaller presentation priority level is output in acomparatively small size. In this way, the management unit 100 mayinclude the presentation priority level determining unit which executesthe function for setting the presentation formats different for thepresentation priority levels of the presentation data items.

As described above, after the presentation data generating unit 302changes the display size of an image data item, a text data item, or avideo data item which is the presentation data item 302 a, thesuperimposing unit 303 superimposes the presentation data item 302 a onthe picture data item 217, and the output unit 108 (the liquid crystaldisplay 511) displays the picture data item 217 with the superimposedpresentation data item 302 a. The presentation data generating unit 302may further change the presentation format of the presentation data item302 a when the user specifies the presentation data item 302 a later.More specifically, the presentation data generating unit 302 may reducea display size of the presentation data item 302 a which is beingdisplayed in a comparatively large size. More specifically, thepresentation data generating unit 302 may increase a display size of thepresentation data item 302 a which is being displayed in a comparativelysmall size. In this way, it is possible to sequentially browse thepresentation data items displayed at the same time while changing thesizes thereof. For example, the user can specify one of the presentationdata items 302 a by touching around the presentation data item 302 adisplayed on the liquid crystal display 511 (touch panel), or performinga pitching operation for widening or narrowing the area including thepresentation data item 302 a on the liquid crystal display 511 (touchpanel).

Here, the demand control request 203 may be a demand request related toan indoor environment parameter indicating a temperature or humidityindoor or an illuminance indicating an amount of light therein. Forexample, the demand requests may be related to indoor environmentparameters indicating “Please set temperature from 24 to 28 degreesCelsius, humidity from 40 to 60 percent, and illuminance from 500 to1000 lux”. The demand requests may be a demand request related only toone environmental parameter indicating a temperature, a humidity, or anilluminance, or may be requests related to any combination of atemperature, a humidity, or an illuminance. When the input unit 105receives a demand request related to an indoor environment parameter,the demand plan generating unit 106 generates a demand plan satisfyingthe demand control request 203 using an energy consuming appliancehaving a function for changing the indoor environment parameter (theenergy consuming appliance is, for example, an air conditioner, ahumidifier, a dehumidifier, a lighting equipment, an electric curtain,or the like). In addition, the appliance observing unit 101 obtains aninformation item related to an indoor environment parameter from anenergy consuming appliance having a function for measuring the indoorenvironment parameter through communication with the energy consumingappliance (the energy consuming appliance is, for example, the airconditioner, the humidifier, the dehumidifier, the lighting equipment,the electric curtain, or the like), and stores the obtained informationitem in a memory area. Next, through the processes as described above,the presentation data item related to the energy consuming appliancehaving the function for measuring the indoor environment parameter isoutput in a superimposed manner. In this way, the presentation data itembased on the demand control request 203 related to the indoorenvironment parameter is output on the output unit 108 in the form ofbeing superimposed around the energy consuming appliance. The user caneasily understand how to use each of the energy consuming appliances soas to satisfy the demand control request 203 related to the indoorenvironment parameter.

Here, the demand control request 203 may be a demand request regardingan air quality in a room in the home. A room air quality is an indicatorof a gas composition, chemical materials, the amount of particles ordusts, a temperature, a humidity, or so on in the air. The demandrequest regarding the room air quality is, for example, “Theconcentration of a material A in the air should be xx ppm.” When theinput unit 105 receives the demand request regarding the room airquality, the demand plan generating unit 106 generates a demand plan forsatisfying the demand control request 203 using the energy consumingappliance having the function for changing the room air quality (theenergy consuming appliance is, for example, an air cleaner, an airconditioner, an air fan, or the like). In addition, the applianceobserving unit 101 obtains an information item regarding the room airquality through communication with the energy consuming appliance havingthe function for changing the room air quality (the energy consumingappliance is, for example, the air cleaner, the air conditioner, an airquality sensor, or the like). Next, through the processes as describedabove, the presentation data item related to the energy consumingappliance having the function for changing the room air quality issuperimposed, and the picture data item is output. In this way, thepresentation data item based on the demand control request 203 relatedto the room air quality is output on the output unit 108 in the form ofbeing superimposed around the energy consuming appliance. The user caneasily understand how to use each of the energy consuming appliances soas to satisfy the demand control request 203 related to the room airquality.

Embodiment 2

An energy management system in this embodiment is characterized bysuperimposing several kinds of presentation data items having differentdisplay levels according to the presentation priority levels of thepresentation data items. In this embodiment, a presentation data itemrelated to an energy consuming appliance having a larger differencebetween a value based on a demand plan and a current value has a higherpresentation priority level to a user.

In this embodiment, the display levels for presentation data items aretwo levels of “detailed display” for presenting detailed information and“brief display” for presenting brief information. The “detailed display”is preferentially assigned to the presentation data item having thehigher presentation priority level to the user.

The basic structure of this embodiment is the same as that ofEmbodiment 1. Thus, the structural elements used in common to those ofEmbodiment 1 are assigned with the same numerical references anddetailed descriptions thereof are omitted where appropriate.

FIG. 6 is a flowchart of the presentation need determining processperformed by a presentation need determining unit 301 in thisembodiment.

First, the presentation need determining unit 301 counts the number N (Nis an integer larger than or equal to 0) of energy consuming appliancesidentified in the picture data item (Step S601).

Next, the presentation need determining unit 301 quantifies thedifferences between values in the demand plan and current values relatedto the identified N energy consuming appliances (Step S602).

Next, the number N of energy consuming appliances is compared with amaximum details presentation number Md (Md is an integer larger than orequal to 1) indicating the number of presentation data items to bepresented at maximum (Step S603). Here, the maximum details presentationnumber Md is an upper limit for the number of presentation data items tobe displayed at the “detailed display” level on the liquid crystaldisplay of the output unit 108 at the same time. In other words, themaximum details presentation number Md is a value preset as a value forsufficiently securing user visibility even when plural presentation dataitems are displayed at the “detailed display” level at the same time,based on the size, resolution, and so on of the liquid crystal display.

When N>Md is satisfied, the number of energy consuming appliancesidentified in the picture data items is larger than the maximum detailspresentation number Md. In this case, Md energy consuming appliances areselected in descending order of the differences ΔE between the values inthe demand plan and the current values calculated in Step S602, and thepresentation data items related to the energy consuming appliances aredetermined to be presentation data items which need to be presented atthe “detailed display” level (Step S604), and a shift is made to StepS606.

When N≤Md is satisfied, the number of energy consuming appliancesidentified in the picture data items is smaller than or equal to themaximum details presentation number Md. For this reason, presentationdata items of N energy consuming appliances identified in the picturedata items are determined to be presentation data items which need to bepresented, for completion of the presentation need determining process(Step S605).

In Step S606, a comparison is made as to (i) a remaining maximumpresentation number according to M−Md (M is an integer larger than orequal to Md and indicates a maximum presentation number) and (ii) aremaining appliance number according to N−Md. Here, the remainingmaximum presentation number according to M−Md is a number obtained bysubtracting the number of presentation data items to be displayed at the“detailed display” level from the number of presentation data itemswhich can be displayed at the “brief display” level.

In addition, the remaining appliance number according to N−Md is thenumber which is (i) obtained by subtracting the number of energyconsuming appliances N identified in the picture data item from thenumber of presentation data items to be displayed at the “detaileddisplay” level, and (ii) indicates the number of energy consumingappliances which are candidates for display at the “brief display”level.

Here, N−Md>M−Md shows that the remaining appliance number according toN−Md is larger than the remaining maximum presentation number accordingto M−Md. In this case, M−Md energy consuming appliances are selectedfrom among N−Md energy consuming appliances in descending order of thedifferences ΔE between the values in the demand plan and the currentvalues calculated in Step S602, and the presentation data items relatedto the energy consuming appliances are determined to be presentationdata items which need to be presented briefly, for completion of thepresentation need determining process (Step S607).

In addition, when N−Md M−Md is satisfied, since the remaining number ofN−Md energy consuming appliances is smaller than or equal to theremaining maximum presentation number according to M−Md, thepresentation data items related to the remaining N−Md energy consumingappliances are determined to be presentation data items which need to bedisplayed briefly, for completion of the presentation need determiningprocess (Step S608).

In the presentation need determining process in this embodiment, it isalso good to determine the needs for presenting the presentation dataitems of only energy consuming appliances having a particular functionsuch as an energy saving mode or energy consuming appliances made by aparticular manufacturer.

In this case, the appliance identifying unit 104 performs a process ofadding appliance identification information items to picture data itemsof the energy consuming appliances having a particular function or theenergy consuming appliances made by the particular manufacturer, withreference to specifications information stored in the applianceinformation storing unit 103.

In addition, the presentation data generating unit 302 in thisembodiment generates presentation data items to be displayed at the“detailed display” level for the presentation data items determined bythe presentation need determining unit 301 as being presentation dataitems which need to be displayed in detail, and generates presentationdata items to be displayed at the “brief display” level for thepresentation data items determined by the presentation need determiningunit 301 as being presentation data items which need to be displayedbriefly.

More specifically, the presentation data items to be displayed at the“detailed display” level include, for example, a text data item whichrequires a comparatively large display area on the liquid crystaldisplay. In contrast, any of presentation data items to be displayed atthe “brief display” level does not require a comparatively large displayarea on the liquid crystal display as required by the text data item.

FIG. 7 shows an example where a picture data item 217 with superimposedpresentation data items is output on the output unit 108 in thisembodiment.

FIG. 7 shows an exemplary case where a liquid crystal display 511 and aspeaker 512 of the mobile terminal 500, and a display application (notshown) and an audio reproduction application (not shown) executed by aCPU of the mobile terminal 500 are configured as the output unit 108.

In the case of FIG. 7 , a maximum presentation number M indicating thenumber of presentation data items to be presented at maximum is 2, and amaximum details presentation number Md is 1. In this case, thepresentation data item for an energy consuming appliance (airconditioner) 152 is determined to be a presentation data item whichneeds to be displayed in detail, and a presentation data item for anenergy consuming appliance (lighting equipment) 153 is determined to bea presentation data item which needs to be displayed briefly.

Here, the energy consuming appliance (air conditioner) 152 has apresentation data item determined to be a presentation data item whichneeds to be presented in detail as having a comparatively largedifference between a value in a demand plan and a current value. Thus,on the liquid crystal display 511, the display application superimposesand displays, as the presentation data items, an image data item 501which visually enhances an image of the energy consuming appliance (airconditioner) 152 and a text data item 502 of “Please increase settingtemperature to 28° C.”.

The energy consuming appliance (lighting equipment) 153 has apresentation data item determined to be a presentation data item whichneeds to be presented briefly. Thus, the display application displaysthe presentation data item such that only an image data item 503 whichvisually enhances an image of the energy consuming appliance (lightingequipment) 153 is superimposed and displayed.

In this way, the presentation data item of the energy consumingappliance having the comparatively large difference between the value inthe demand plan and the current value is superimposed as having a highdisplay level. In this way, it is possible to increase visibility of thepresentation data item which (i) is related to the energy consumingappliance having the comparatively large difference between the value inthe demand plan and the current value, and (ii) indicates control forproducing a comparatively large effect when performed.

For this reason, it is expected that the user visually recognizes, onthe output unit 108, the picture data item with the superimposedpresentation data item to be output by the output unit 108, andexecutes, on the presentation data item, comparatively effective controlfor achieving the demand control request in a more reliable manner.

Accordingly, it is possible to prompt the user to perform thecomparatively effective method for using the energy consuming applianceto achieve the demand control request.

The display levels for the presentation data items are two in the abovedescription, but three or more display levels may be used.

Embodiment 3

An energy management system in this embodiment is characterized byincluding a function for automatically controlling an energy consumingappliance based on a demand plan, and superimposing presentation dataitems such that an energy consuming appliance to be automaticallycontrolled and an energy consuming appliance not to be automaticallycontrolled are distinguished from each other.

The basic structure of this embodiment is largely in common to that ofEmbodiment 1. Thus, the structural elements used in common to those ofEmbodiment 1 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

FIG. 8 is a diagram showing a configuration of the energy managementsystem in this embodiment.

The energy management system in this embodiment includes an appliancecontrol unit 110 and a plan executing unit 109, in addition to thesystem configuration in Embodiment 1.

The appliance control unit 110 controls the energy consuming appliancethrough wired or wireless communication with the energy consumingappliance. The plan executing unit 109 controls the execution of thedemand plan by the energy consuming appliance to be automaticallycontrolled through the appliance control unit 110.

The appliance control unit 110 does not always automatically control allof n energy consuming appliances 151 to 15 n through communication withthe appliances. It is assumed that a user has in his or her home (i) anenergy consuming appliance which can be operated only manually by theuser and (ii) an energy consuming appliance which can be controlled bythe appliance control unit 110 but is set so as not to be automaticallycontrolled.

In this embodiment, information items indicating whether the respectiveenergy consuming appliances can be controlled, or are subject toautomatic control are stored in the appliance information storing unit103 as specifications information items.

In addition, the input unit 105 in this embodiment receives aninstruction for controlling one of the energy consuming appliances inputby the user in addition to the demand control request.

The instruction for controlling the energy consuming appliance inputthrough the input unit 105 is received by the appliance control unit110. The appliance control unit 110 controls the energy consumingappliance based on the received control instruction.

The aforementioned plan executing unit 109 and the appliance controlunit 110 are configured actually in the forms of one or more CPUs and aplurality of memory appliances, in addition to the configurationincluding the appliance observing unit 101, the appliance informationstoring unit 103, the appliance identifying unit 104, the input unit105, the demand plan generating unit 106, and the smart informationsuperimposing unit 107, and the output unit 108.

By means of the CPU(s) executing the program stored in the memoryappliance and storing the information in the format which allowsreading-out onto the memory area on the memory appliance, the functionsof the plan executing unit 109 and the appliance control unit 110 arerealized together with the functions of the appliance observing unit101, the appliance information storing unit 103, the applianceidentifying unit 104, the demand plan generating unit 106, and the smartinformation superimposing unit 107.

FIG. 9 is a diagram showing a sequence of operations performed by theenergy management system in this embodiment.

A sequence from an appliance observing process 201 by the applianceobserving unit 101 to a process in which the output unit 108 outputspicture data item 217 with superimposed presentation data items using aliquid crystal display and/or a speaker is similar to the one inEmbodiment 1.

In this embodiment, the demand plan generating unit 106 transmits thegenerated demand plan 221 to the plan executing unit 109, in addition totransmitting the presentation information items 207 to the smartinformation superimposing unit 107.

At this time, the demand plan generating unit 106 transmits the demandplan 221 related to an energy consuming appliance which has been set soas to be automatically controlled based on the demand plan.

Upon receiving the demand plan 221, the plan executing unit 109transmits the control instruction 223 to the appliance control unit 110so that the energy consuming appliance operates based on the details ofthe demand plan 221 (a plan executing process 222).

The appliance control unit 110 transmits the control signal 225corresponding to the received control instruction 223 to each of theenergy consuming appliances, and controls the energy consuming appliance(an appliance control process 224).

In this way, in the case of the energy consuming appliance which isautomatically controlled based on the demand plan, the differencebetween the state in the demand plan and an actual operation state iscomparatively small.

In other words, in the case of the energy consuming appliance subject toautomatic control is to be automatically controlled based on the demandplan. For this reason, compared to the energy consuming appliance whichis not subject to the automatic control, the energy consuming appliancesubject to the automatic control does not comparatively need to benotified to the user in terms of the effective use.

For this reason, the smart information superimposing unit 107 in thisembodiment preferentially superimposes a presentation data item for theenergy consuming appliance which is not subject to the automatic controlover a presentation data item for energy consuming appliance which issubject to the automatic control.

The smart information superimposing unit 107 in this embodiment has aninternal structure similar to the one in Embodiment 1. However, thepresentation need determining process by the presentation needdetermining unit 301 and the presentation data generating process by thepresentation data generating unit 302 are characteristic with respect tothose in the earlier-described embodiments.

FIG. 10 is a flowchart of a presentation need determining process by thepresentation need determining unit 301 in this embodiment.

First, the presentation need determining unit 301 separately counts (i)the number Nm (Nm is an integer larger than or equal to 0) of energyconsuming appliances which are not subject to the automatic control and(ii) the number Na (Na is an integer larger than or equal to 0) ofenergy consuming appliances which are subject to the automatic control,among the energy consuming appliances identified in the picture dataitem (Step S1001).

Next, the presentation need determining unit 301 quantifies thedifferences between values in the demand plan and current values relatedto the respectively identified energy consuming appliances (Step S1002).

Next, the number Nm of energy consuming appliances is compared with themaximum presentation number M (M is an integer larger than or equalto 1) indicating the number of presentation data items to be presentedat maximum (Step S1003).

When Nm>M is satisfied, the number Nm of energy consuming appliancesidentified in the picture data item as being not subject to theautomatic control is larger than the maximum presentation number M. Inthis case, M energy consuming appliances are selected from among theenergy consuming appliances identified in the picture data item as beingnot subject to the automatic control. In other words, the M energyconsuming appliances are selected in descending order of the differencesΔE between the values in the demand plan and the current valuescalculated in Step S1002. For this reason, presentation data items of Menergy consuming appliances are determined to be presentation data itemswhich need to be presented, for completion of the presentation needdetermining process (Step S1004).

In addition, when Nm M is satisfied, the number Nm of energy consumingappliances identified in the picture data item as being not subject tothe automatic control is smaller than or equal to the maximumpresentation number M. Thus, presentation data items for the Nm energyconsuming appliances identified in the picture data item are determinedto be presentation data items which need to be presented (Step S1005).

Next, the presentation need determining unit 301 selects M−Nm energyconsuming appliances at maximum in descending order of the differencesΔE between the values in the demand plan and the current valuescalculated in Step S1002 from among Na energy consuming appliances whichare subject to the automatic control, and determines the selected M−Nmenergy consuming appliances to be appliances for which presentation dataitems need to be presented, for completion of the presentation needdetermining process (Step S1006).

As in Embodiment 1, the presentation data generating unit 302 in thisembodiment generates presentation data items including image data items,audio data items, text data items, video data items, etc. It is to benoted that the presentation data items are generated to have differentformats distinguishable between the energy consuming appliances whichare subject to the automatic control and the energy consuming applianceswhich are not subject to the automatic control. More specifically,presentation data items including icon images showing that the automaticcontrol is being executed are generated for the energy consumingappliances subject to the automatic control.

FIG. 11 is a diagram showing an exemplary output by the output unit 108in this embodiment.

FIG. 11 shows an exemplary case where a liquid crystal display 511 and aspeaker 512 of the mobile terminal 500, and a display application (notshown) and an audio reproduction application (not shown) executed on aCPU of the mobile terminal 500 are configured as the output unit 108.

FIG. 11 shows an example where the maximum presentation number Mindicating the number of presentation data items to be presented atmaximum is 2, the energy consuming appliance (television receiver) 151and, the energy consuming appliance (lighting equipment) 153 are subjectto the automatic control, and the energy consuming appliance (airconditioner) 152 is not subject to the automatic control. This exampleshows the case where presentation data items for the energy consumingappliance (air conditioner) 152 and the energy consuming appliance(lighting equipment) 153 are determined to be presentation data itemswhich need to be presented.

In this case, on the liquid crystal display 511, the display applicationsuperimposes and displays an icon image 1103 showing that the automaticcontrol is currently being executed, as the presentation data item forthe energy consuming appliance (lighting equipment) 153. Furthermore,data items such as an image data item 501 which visually enhances theenergy consuming appliance (air conditioner) 152 and a text data item502 of “Please increase setting temperature to 28° C.!” are superimposedand displayed as presentation data items for the energy consumingappliance (air conditioner) 152.

In addition, when the user taps the energy consuming appliance (lightingequipment) 153 on the touch panel integrated with the liquid crystaldisplay 511, the audio reproduction application outputs a speech of“Under automatic control!” from the speaker 512.

In addition, when the user taps the energy consuming appliance (airconditioner) 152, the audio reproduction application outputs a speech of“Please increase setting temperature to 28° C.!” from the speaker 512.

In this embodiment, the presentation data item for the energy consumingappliance which is not subject to the automatic control is superimposedpreferentially over the presentation data items for the energy consumingappliances which are subject to the automatic control. In this case, thepresentation data item for the energy consuming appliance which is notsubject to the automatic control is visually recognized by the user.Thus, effective control based on the presentation data item is performedto achieve the demand control request. In this way, it is expected thatthe operation for control related to the presentation data item isexecuted by the user in a more reliable manner.

In this embodiment, presentation data items are generated to havedifferent formats distinguishable between the energy consumingappliances subject to the automatic control and the energy consumingappliance not subject to the automatic control, for example, as shown bythe icon image 1103 indicating that the automatic control is currentlybeing executed.

In this case, the user can not only distinguish the energy consumingappliances which are subject to the automatic control and the energyconsuming appliance which is not subject to the automatic control, butalso can be relieved that the energy consuming appliances subject to theautomatic control are being automatically controlled based on the demandplan.

Here, the superimposing unit 303 in this embodiment may superimpose anactivation button for activating a remote control application on thepicture data item 215 with identification information items, togetherwith the presentation data items.

In this case, the remote control application is an application forcontrolling energy consuming appliances remotely. When the activationbutton superimposed on the picture data item is pressed by the user, aremote control operation for a corresponding energy consuming applianceis activated.

The remote control operation may be input through a menu display screenon the touch panel integrated with the liquid crystal display 511, butpreferably be input in the form of a speech through the audio microphoneof the input unit 105.

The control instructions recognized by the speech recognitionapplication of the input unit 105 are transmitted to the appliancecontrol unit 110. Upon receiving the control instructions, the appliancecontrol unit 110 controls the energy consuming appliances based on thecontrol instructions.

In this case, the user can control the energy consuming applianceremotely while watching the display screen with the superimposedpresentation data items based on the demand plan. In other words, theuser can perform, on energy consuming appliances, remote controloperations which include adjusting a set temperature or switchingoperation modes for the air conditioner, adjusting the intensity of thelight, switching ON or OFF of any of the energy consuming appliances,etc., resulting in increase in the userfriendliness.

Embodiment 4

Compared to Embodiment 3, an energy management system in this embodimentis further characterized by superimposing several kinds of presentationdata items having different display levels according to the presentationpriority levels of the presentation data items. In this embodiment, apresentation data item related to an energy consuming appliance having alarger difference between a value based on a demand plan and a currentvalue has a higher presentation priority level to a user.

In this embodiment, the display levels for presentation data items aretwo levels of “detailed display” for presenting detailed information and“brief display” for presenting brief information. The former “detaileddisplay” is preferentially assigned to a presentation data item having ahigher presentation priority level to the user.

However, this embodiment is the same in the basic structure asEmbodiment 3, and thus the structural elements used in common to thoseof Embodiment 3 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

Each of FIGS. 12A, 12B, and 12C is a diagram showing a part of theflowchart of the presentation need determining process performed by thepresentation need determining unit 301 in this embodiment.

First, the presentation need determining unit 301 separately counts (i)the number Nm (Nm is an integer larger than or equal to 0) of energyconsuming appliances which are not subject to the automatic control and(ii) the number Na (Na is an integer larger than or equal to 0) ofenergy consuming appliances which are subject to the automatic control,among the energy consuming appliances identified in the picture dataitem (Step S1201).

Next, the presentation need determining unit 301 quantifies thedifferences between values in the demand plan and current values relatedto the respectively identified energy consuming appliances (Step S1202).

Next, the number Nm of energy consuming appliances which are not subjectto the automatic control is compared with the maximum presentationnumber Md (Md is an integer larger than or equal to 1) indicating thenumber of presentation data items to be presented at maximum (StepS1203).

When Nm>Md is satisfied, the number of energy consuming appliancesidentified in the picture data item is larger than the maximum detailspresentation number Md, and thus Md energy consuming appliances areselected. In other words, from among the energy consuming appliancesidentified in the picture data item, the Md energy consuming appliancesare selected in descending order of the differences ΔE between thevalues in the demand plan and the current values calculated in StepS1202. The presentation data items related to the Md energy consumingappliances are determined to be presentation data items which need to bepresented in detail (Step S1204), and a shift is made to Step S1206.

When Nm≤Md is satisfied, the number of energy consuming appliances whichare not subject to the automatic control among the energy consumingappliances identified in the picture data item is smaller than or equalto the maximum details presentation number Md. Thus, the presentationdata items related to the Nm energy consuming appliances which are notsubject to the automatic control are determined to be presentation dataitems which need to be presented in detail (Step S1205), and a shift ismade to Step S1210.

In Step S1206, a comparison is made as to (i) a remaining maximumpresentation number according to M−Md (M is an integer larger than orequal to Md and indicates a maximum presentation number) and (ii) aremaining appliance number according to Nm−Md, indicating the number ofremaining energy consuming appliances which are not subject to theautomatic control.

Here, the remaining maximum presentation number according to M−Md is anumber obtained by subtracting the number of presentation data items tobe displayed at the “detailed display” level from the number ofpresentation data items which can be displayed at the “brief display”level.

In addition, the remaining appliance number according to Nm−Md is anumber obtained by subtracting the number of energy consuming appliancesto be displayed at the “detailed display” level from the number Nm ofenergy consuming appliances which are not subject to the automaticcontrol. In other words, the value obtained according to Nm−Md shows thenumber of energy consuming appliances which are candidates to bedisplayed at the “brief display” level.

Here, Nm−Md>M−Md shows that the remaining appliance number according toNm−Md is larger than the remaining maximum presentation number accordingto M−Md. For this, M−Md energy consuming appliances are selected indescending order of the differences ΔE between the values in the demandplan and the current values calculated in Step S1202 from among theremaining Nm−Md energy consuming appliances. For this reason,presentation data items related to the selected energy consumingappliances are determined to be presentation data items which need to bepresented briefly, for completion of the presentation need determiningprocess (Step S1207).

In addition, when Nm−Md M−Md is satisfied, since the number of remainingN−Md energy consuming appliances is smaller than or equal to theremaining maximum presentation number according to M−Md, thepresentation data items related to the remaining Nm−Md energy consumingappliances are determined to be presentation data items which need to bedisplayed briefly (Step S1208).

Next, the presentation need determining unit 301 selects M−Nm energyconsuming appliances at maximum in descending order of the differencesΔE between the values in the demand plan and the current valuescalculated in Step S1202 from among Na energy consuming appliances whichare subject to the automatic control, and determines the selected M−Nmenergy consuming appliances to be appliances for which presentation dataitems need to be presented, for completion of the presentation needdetermining process (Step S1209).

In Step S1210, the number Na of energy consuming appliances subject tothe automatic control is compared with the remaining maximum detailspresentation number according to Md−Nm (Md is an integer larger than orequal to 1) for the presentation data items.

When Na>Md−Nm is satisfied, the number of energy consuming appliancesidentified in the picture data item as being subject to the automaticcontrol is larger than the remaining maximum details presentation numberaccording to Md−Nm. In this way, the Md−Nm energy consuming appliancesare selected in descending order of the differences ΔE between thevalues in the demand plan and the current values calculated in StepS1202. The presentation data items related to the Md−Nm energy consumingappliances are determined to be presentation data items which need to bepresented in detail (Step S1211), and a shift is made to Step S1213.

In addition, when Na≤Md−Nm is satisfied, the number of energy consumingappliances identified in the picture data item as being subject to theautomatic control is smaller than or equal to the remaining maximumdetails presentation number according to Md−Nm. For this reason,presentation data items related to Na energy consuming appliancesidentified in the picture data item are determined to be presentationdata items which need to be presented in detail, for completion of thepresentation need determining process (Step S1212).

In Step S1213, a comparison is made as to (i) a remaining maximumpresentation number according to M−Md for the presentation data items (Mis an integer larger than or equal to Md and indicates a maximumpresentation number) and (ii) a remaining appliance number according toN−Md.

Here, N−Md>M−Md shows that the remaining appliance number according toN−Md is larger than the remaining maximum presentation number accordingto M−Md. For this, M−Md energy consuming appliances are selected indescending order of the differences ΔE between the values in the demandplan and the current values calculated in Step S1202 from among theremaining N−Md energy consuming appliances subject to the automaticcontrol. For this reason, presentation data items related to the M−Mdenergy consuming appliances are determined to be presentation data itemswhich need to be presented briefly, for completion of the presentationneed determining process (Step S1214).

In addition, when N−Md M−Md is satisfied, since the remaining number ofN−Md energy consuming appliances is smaller than or equal to theremaining maximum presentation number according to M−Md, thepresentation data items related to the remaining N−Md energy consumingappliances are determined to be presentation data items which need to bedisplayed briefly, for completion of the presentation need determiningprocess (Step S1215).

As in Embodiment 1, the presentation data generating unit 302 in thisembodiment generates presentation data items including image data items,audio data items, text data items, video data items, etc.

It is to be noted that the presentation data items are generated to havedifferent formats distinguishable between the energy consumingappliances which are subject to the automatic control and the energyconsuming appliances which are not subject to the automatic control.

In addition, the presentation data generating unit 302 in thisembodiment generates presentation data items to be displayed at the“detailed display” level for the presentation data items determined bythe presentation need determining unit 301 as being presentation dataitems which need to be displayed in detail, and generates presentationdata items to be displayed at the “brief display” level for thepresentation data items determined by the presentation need determiningunit 301 as being presentation data items which need to be displayedbriefly.

FIG. 13 shows an example where a picture data item 217 with superimposedpresentation data items is output on the output unit 108 in thisembodiment.

FIG. 13 shows an example where a liquid crystal display 511 and aspeaker 512 of the mobile terminal 500, and the display application (notshown) and the audio reproduction application (not shown) executed onthe CPU of the mobile terminal 500 are configured as the output unit108.

In the case of FIG. 13 , the maximum presentation number M indicatingthe maximum number of presentation data items is 3, the maximum detailspresentation number Md is 2, and the energy consuming appliance(television receiver) 151, and the energy consuming appliance (lightingequipment) 153 are subject to the automatic control. In this examplecase, the energy consuming appliance (air conditioner) 152 is notsubject to the automatic control, the presentation data items for theenergy consuming appliance (air conditioner) 152 and the energyconsuming appliance (lighting equipment) 153 are determined to bepresentation data items which need to be presented in detail, and thepresentation data item for the energy consuming appliance (televisionreceiver) 151 is determined to be a presentation data item which needsto be presented briefly.

As for the energy consuming appliance (air conditioner) 152 which is notsubject to the automatic control and for which the presentation dataitem is determined to be presented in detail, a presentation data itemis superimposed and displayed on the liquid crystal display 511. Forexample, an image data item 501 which visually enhances the energyconsuming appliance (air conditioner) 152 and a text data item 502 of“Please increase setting temperature to 28° C.!” are superimposed anddisplayed as presentation data items on the liquid crystal display 511.

In addition, as for the energy consuming appliance (lighting equipment)153 which is subject to the automatic control and for which thepresentation data item is determined to be presented in detail, an iconimage 1103 showing that the automatic control is being executed and atext data item 502 of “Under automatic control! Energy saved by 15%” aresuperimposed and displayed as presentation data items on the liquidcrystal display 511.

In addition, as for the energy consuming appliance (television receiver)151 which is subject to the automatic control and for which thepresentation data item is determined to be presented briefly, an iconimage 1104 showing that the automatic control is being executed issuperimposed and displayed on the liquid crystal display 511.

In this embodiment, the presentation data items for the energy consumingappliance which is not subject to the automatic control is superimposedpreferentially over the presentation data items for the energy consumingappliances which are subject to the automatic control. Furthermore, thepresentation data item of the energy consuming appliance having thecomparatively large difference between the value in the demand plan andthe current value is superimposed as having a high display level. Inthis way, it is possible to increase visibility of the presentation dataitem, among the presentation data items, which (i) is related to theenergy consuming appliance having the comparatively large differencebetween the value in the demand plan and the current value, and (ii)indicates control for producing a comparatively large effect whenperformed.

For this reason, the presentation data item superimposed on the picturedata item output on the output unit 108 is visually recognized by theuser, and thus it is expected that the operation for control related tothe presentation data item is executed by the user in a more reliablemanner.

Accordingly, it is possible to prompt the user to perform thecomparatively effective method for using the energy consuming applianceto achieve the demand control request.

The display levels for the presentation data items are two in the abovedescription, but three or more display levels may be used.

Embodiment 5

An energy management system in this embodiment is characterized in thatpresentation data items based on a demand plan can be superimposed andseen in a picture data item obtained by imaging energy consumingappliances even when the energy consuming appliances are imaged by acamera unit 102 which does not face the energy consuming appliances.

The structural elements of this embodiment are largely in common tothose of Embodiment 3, and thus the common structural elements areassigned with the same numerical references, and indication of diagramsand detailed descriptions thereof are omitted where appropriate.

FIG. 14 is a diagram showing a configuration of an energy managementsystem in this embodiment.

The energy management system in this embodiment includes n energyconsuming appliances 151 to 15 n, and three units of a home managementunit 1401, a mobile terminal management unit 1402, and a servermanagement unit 1403. The management unit 100 which manages energyconsumed by the energy consuming appliances 151 to 15 n includes theaforementioned three units.

The home management unit 1401 includes a communication unit 1411 whichcommunicates with each of the following appliances placed in a home: anappliance observing unit 101; a plan executing unit 109; an appliancecontrol unit 110; the mobile terminal management unit 1402; and theserver management unit 1403.

The mobile terminal management unit 1402 includes a communication unit1412 which communicates with each of the following mobile terminalshaving structures which allow the user to carry around: the camera unit102; an appliance identifying unit 104; an input unit 105; a smartinformation superimposing unit 107; an output unit 108; the homemanagement unit 1401; and the server management unit 1403.

The server management unit 1403 includes, as a computer serverappliance, a communication unit 1413 which communicates with each of anappliance information storing unit 103, a demand plan generating unit106, the home management unit 1401, and the mobile terminal managementunit 1402. The management unit 100 is an example of a specific versionof the management unit 100 in FIG. 8 .

The communication units 1411, 1412, and 1413 are connected to each otherso as to be communicable wired or wirelessly.

FIG. 15 shows an internal structure of the smart informationsuperimposing unit 107 in this embodiment.

The smart information superimposing unit 107 in this embodimentincludes: not only a presentation need determining unit 301; apresentation data generating unit 302; and a superimposing unit 303 butalso processing units below.

The processing units are: an image data storing unit 1504 which storespicture and picture items to which appliance identification informationitems are added by the appliance identifying unit 104; and a switchingunit 1505 which switches picture data items on which presentation dataitems are to be superimposed, by switching picture data items to beinput to the presentation need determining unit 301, the presentationdata generating unit 302, and the superimposing unit 303.

The image data storing unit 1504 stores the picture and image dataitems. A picture data item is obtained by imaging one or more energyconsuming appliances in a room of the home using the camera unit 102,and an image data item is obtained by imaging each of one or more energyconsuming appliances in the home using the camera unit 102. The pictureand image data items are stored in advance with the applianceidentification information items added by the appliance identifying unit104.

Here, each of attribute information items of the picture and image dataitems stored in the image data storing unit 1504 includes a positioninformation item indicating a place at which the image is imaged. Thisposition information item is obtained using a position informationobtaining function such as a Global Positioning System (GPS). Theposition information obtaining function is not shown in FIG. 14 , but isprovided to the mobile terminal included in the mobile terminalmanagement unit 1402.

The switching unit 1505 reads out a picture or image data item stored inthe image data storing unit 1504, and receives the picture data itemwith the appliance identification information items transmitted from theappliance identifying unit 104. The switching unit 1505 switches to theaforementioned picture data item to be output to the presentation needdetermining unit 301, the presentation data generating unit 302, and thesuperimposing unit 303, based on a switch instruction 1615 input by theuser through the input unit 105. In other words, the switch instruction1615 includes a control mode which is information for selecting apicture or image data item to be output onto the presentation needdetermining unit 301 etc. from among the selection candidates includingthe picture or image data item stored in the image data storing unit1504.

Here, the switch instruction 1615 may further include, as a controlmode, information for specifying either a picture data item stored inthe image data storing unit 1504 is output to the presentation needdetermining unit 301 etc. or a picture data item etc. imaged by thecamera unit 102 is output to the presentation need determining unit 301etc. For example, when the switch instruction 1615 includes a “storedimage mode” as the control mode, the picture data item stored in theimage data storing unit 1504 is output to the presentation needdetermining unit 301 etc. In addition, when the switch instruction 1615includes a “camera mode” as the control mode, the picture data item etc.imaged by the camera unit 102 is output to the presentation needdetermining unit 301 etc.

For example, the switching unit 1505 may switch to picture data itemswith appliance identification information items stored in the image datastoring unit 1504. In this case, the picture data item which is obtainedby imaging the energy consuming appliances placed in the home and isstored in the image data storing unit 1504 includes presentation dataitems based on the demand plan superimposed thereon. In this case, evenwhen the camera unit 102 does not always face the energy consumingappliances, the superimposed presentation data items based on the demandplan can be seen on the picture data item obtained by imaging the actualenergy consuming appliances.

On the other hand, as in the case of Embodiment 3, when a switch is madeto the picture data item with the appliance identification informationitems transmitted from the appliance identifying unit 104, thepresentation data items based on the demand plan are superimposed andseen on the picture data item obtained by imaging the energy consumingappliances which are currently being followed by the camera unit 102.

FIG. 16 is a sequence diagram showing a sequence of operations performedby the energy management system in this embodiment.

The following sequences are similar to those in Embodiment 3: a sequenceof processes from an appliance observing process 201 by the applianceobserving unit 101 to when a control signal 225 based on the demand planis transmitted; and a sequence from an image capturing process 211 bythe camera unit 102 and to when the picture data item 215 with theidentification information items is transmitted to the smart informationsuperimposing unit 107.

In this embodiment, the switch instruction 1615 for switching to thepicture data item on which a presentation data item is to besuperimposed is input by the user through the input unit 105 (an inputprocess 1602), the switch instruction 1615 which is an information itemindicating the details of the input is transmitted to the smartinformation superimposing unit 107.

Upon receiving the switch instruction 1615, the smart informationsuperimposing unit 107 switches the picture data item to be output tothe presentation need determining unit 301, the switching unit 1505switches to the picture data item to be output to the presentation datagenerating unit 302, and the superimposing unit 303, based on the switchinstruction 1615.

The smart information superimposing unit 107 performs a smartsuperimposing process 1616 similar to the one in Embodiment 3 onto thepicture data item after the switch by the switching unit 1505.

The smart information superimposing unit 107 performs the smartsuperimposing process 1616, and transmits the picture data item 1617with the superimposed presentation data item to the output unit 108.

The output unit 108 outputs the picture data item 1617 with thesuperimposed presentation data item using the liquid crystal display andthe speaker (an output process 1618).

Each of FIGS. 17A, 17B, and 17C is a diagram showing an example wherethe picture data item 1617 with the superimposed presentation data itemin this embodiment is output on the liquid crystal display 511 of theoutput unit 108.

FIG. 17A shows a case in which the switching unit 1505 switches to thepicture data item on which the presentation data item is to besuperimposed in a living room, based on the switch instruction 1602input through the input unit 105.

In the case of FIG. 17A, the energy consuming appliance (lightingequipment), the energy consuming appliance (air conditioner), and theenergy consuming appliance (television receiver) are displayed on theliquid crystal display on the output unit 108.

Furthermore, as for the energy consuming appliance (air conditioner), animage data item 1101 which visually enhances the energy consumingappliance (air conditioner) and a text data item 1102 of “Pleaseincrease setting temperature to 28° C.!” are superimposed and displayedas presentation data items.

Furthermore, as for energy consuming appliance (lighting equipment), anicon image 1103 showing that an automatic control is being executed aresuperimposed and displayed as presentation data items.

In addition, FIG. 17B shows a case where the switching unit 1505switches to a picture data item obtained by imaging a bedroom 1 as apicture data item to which a presentation data item is superimposed,based on the switch instruction 1602 input through the input unit 105.

In the case of FIG. 17B, the energy consuming appliance (lightingequipment), the energy consuming appliance (air conditioner), and theenergy consuming appliance (television receiver) in the bedroom 1 aredisplayed on the liquid crystal display on the output unit 108.

Furthermore, as for the energy consuming appliance (air conditioner), animage data item 1101 which visually enhances the energy consumingappliance (air conditioner) and a text data item 1102 of “Pleaseincrease setting temperature to 28° C.!” are superimposed and displayedas presentation data items.

Furthermore, as for energy consuming appliance (lighting equipment), anicon image 1103 showing that an automatic control is being executed aresuperimposed and displayed as presentation data items.

In addition, FIG. 17C shows a case where the switching unit 1505switches to a picture data item obtained by imaging a bedroom 2 as apicture data item to which a presentation data item is superimposed,based on the switch instruction 1602 input through the input unit 105.

In the case of FIG. 17C, the energy consuming appliance (lightingequipment), and the energy consuming appliance (air conditioner) in thebedroom 2 are displayed on the liquid crystal display on the output unit108.

Furthermore, as for the energy consuming appliance (air conditioner), animage data item 1101 which visually enhances the energy consumingappliance (air conditioner) and a text data item 1102 of “Pleaseincrease setting temperature to 28° C.!” are superimposed and displayedas presentation data items.

Furthermore, as for energy consuming appliance (lighting equipment), anicon image 1103 showing that an automatic control is being executed aresuperimposed and displayed as presentation data items.

Here, the switch instruction 1615 may be input in association with aflick operation on the touch panel through the input unit 105.

In this case, in response to the flick operation, the switching unit1505 sequentially switches to picture data items to be output to thepresentation need determining unit 301, the presentation data generatingunit 302, and the superimposing unit 303.

In this way, the user can switch the picture data items on whichpresentation data items are superimposed in a comparatively easy mannerby performing operations similar to the operations for switchingphotograph data items to be displayed on the liquid crystal display 511of the mobile terminal 500.

As described above, in this embodiment, the presentation data itemsbased on the demand plan are superimposed and seen in the picture dataitems obtained by imaging the energy consuming appliances even when thecamera unit 102 does not always face the energy consuming appliancesduring the imaging.

In other words, the user can see the presentation data items based onthe demand plan superimposed on the imaged data items of the energyconsuming appliances in each room without going to the room.

In addition, the user can see the presentation data items based on thedemand plan superimposed on the image data items of the energy consumingappliances outside the home, with an increased userfriendliness.

In addition, when the mobile terminal 500 is set to a mode fordisplaying the picture data items on which presentation data items aresuperimposed, the following operation may be performed. The operationwhich may be performed here is to output a picture data item with (i)the energy consuming appliance having the largest difference between avalue in the demand plan and a current value and (ii) a presentationdata item superimposed on the picture data item.

In this case, the smart information superimposing unit 107 identifiesthe energy consuming appliance having the largest difference between avalue in the demand plan and a current value, based on the demand planincluded in the presentation information items and the appliance stateinformation items. The switching unit 1505 performs the switchoperations to the picture data items including the identified energyconsuming appliances.

In this way, the presentation data item is firstly performed which (i)is related to the energy consuming appliance having the largestdifference between the value in the demand plan and the current value,and (ii) provides the highest effect when control based on thepresentation data item is performed. In this case, the operation forcontrol which is related to the presentation data item and effective forachieving the demand control request is visually recognized by the user,and thus it is expected that the operation is executed by the user in amore reliable manner.

In addition, the switching unit 1505 switches picture data items basedon the switch instruction 1602 input through the input unit 105 in theabove description, the switching unit 1505 may automatically switch thepicture data items based on time information items or positioninformation items.

For example, in the case of a switch based on time information items,the switching unit 1505 sequentially switches, at a predetermined timeinterval, picture data items to be output to the presentation needdetermining unit 301, the presentation data generating unit 302, and thesuperimposing unit 303.

In this way, the picture data items on which presentation data items aresuperimposed stored in the image data storing unit 1504 are to be outputfrom the output unit 108 in a manner like a slide show.

In addition, for example, in the case of a switch based on positioninformation items, the switching unit 1505 automatically switches to apicture data item imaged at a place closest to a current user positionbased on a user position information item, as a picture data itemdetermined to be output to the presentation need determining unit 301,the presentation data generating unit 302, and the superimposing unit303.

At this time, the switching unit 1505 compares position informationitems included in attribute information items of the image data itemsstored in the image data storing unit 1504, and switches to the imagedata item closest to the current user position based on the userposition information item (the user here carries around the mobileterminal 500).

In this case, for example, when the user moves from the living room toone of the bedrooms, the picture data item imaged in the living room isswitched to the picture data item imaged in the bedroom, and the picturedata item imaged in the bedroom and presentation data items superimposedon the picture data item are output on the output unit 108.

Here, the superimposing unit 303 in this embodiment may superimpose anactivation button of a remote control application together with thepresentation data items, on the picture data item 1504 withidentification information items.

In this case, when the activation button superimposed on the picturedata item is pressed by the user, the remote control operation relatedto the corresponding energy consuming appliance is activated.

The remote control operation may be input through a menu display screenon the touch panel integrated with the liquid crystal display 511, butpreferably be input in the form of a speech through the audio microphoneof the input unit 105.

The control instruction recognized by the speech recognition applicationof the input unit 105 is transmitted to the appliance control unit 110.Upon receiving the control instruction, the appliance control unit 110controls the energy consuming appliance based on the controlinstruction.

In this case, the user outside can control the energy consumingappliance remotely while watching the display screen with thesuperimposed presentation data item based on the demand plan.

In other words, the user can perform, on the energy consumingappliances, remote control operations which include adjusting a settemperature or switching operation modes for the air conditioner,adjusting the intensity of the light, switching ON or OFF of any of theenergy consuming appliances, etc., resulting in increase in theuserfriendliness.

Embodiment 6

An energy management system in this embodiment is characterized bygenerating a plurality of demand plans, and superimposing presentationdata items based on one of the demand plans selected by a user.

The basic structure of this embodiment is largely in common to that ofEmbodiment 3. Thus, the structural elements used in common to those ofEmbodiment 3 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

FIG. 18 is a diagram showing a configuration of an energy managementsystem in this embodiment.

In this embodiment, the demand plan generating unit 1806 generates thedemand plans.

The demand plan generating unit 1806 generates the demand plans, forexample, based on a method performed by a simulation executing unit togenerate reduction plans described in paragraphs [0078] to [0106] ofPatent Literature 2.

The demand plan generating unit 1806 performs a simulation for acombination of a current appliance configuration and each of controlpatterns, evaluates the result of the simulation, derives some of thecontrol patterns which will achieve the demand request (corresponding tothe reduction target in Patent Literature 2). The details of the derivedcontrol patterns are regarded as the demand plans.

The demand plan generating unit 1806 generates presentation informationitems including the generated demand plans, and transmits them to thesmart information superimposing unit 1807.

When the operation for selecting one of the demand plans is performed bythe user, the selection instruction indicating the selected demand planis transmitted to the smart information superimposing unit 1807 throughthe input unit 105.

The smart information superimposing unit 1807 selects the demand plancorresponding to the demand plan indicated by the selection instructionreceived through the input unit 105, and superimposes the presentationdata items based on the selected demand plan onto a picture data itemobtained by imaging energy consuming appliances in user's home.

In addition, the demand plans generated by the demand plan generatingunit 1806 are also transmitted to the plan executing unit 1809.

In addition, the selection instruction indicated by the selected demandplan is also transmitted to the plan executing unit 1809 through theinput unit 105. The plan executing unit 1809 executes the demand planindicated by the selection instruction received through the input unit105.

FIG. 19 shows an internal structure of the smart informationsuperimposing unit 1807 in this embodiment.

The smart information superimposing unit 1807 in this embodimentincludes a presentation need determining unit 301, a presentation datagenerating unit 302, a superimposing unit 303, and a demand planselecting unit 1901 for selecting a demand plan to be superimposed onthe picture data item 215 with identification information items.

Based on the selection instruction 1051 received through the input unit105, the demand plan selecting unit 1901 selects the demand plan to besuperimposed on the picture data item 215 with identificationinformation items, from among the demand plans included in thepresentation information items received from the demand plan generatingunit 1806.

FIG. 20 is a diagram showing a sequence of operations performed by theenergy management system in this embodiment.

The following sequences are similar to those in Embodiment 3: a sequenceof processes from an appliance observing process 201 by the applianceobserving unit 101 to when appliance specification information items 205are transmitted to the demand plan generating unit 1806; and a sequencefrom an image capturing process 211 by the camera unit 102 to when thepicture data item 215 with the identification information items istransmitted to the smart information superimposing unit 107.

In this embodiment, the demand plan generating unit 1806 generates thedemand plans (a demand plan generating process 2006), transmits thepresentation information items 2007 including the generated demand plansto the smart information superimposing unit 1807, and transmits thegenerated demand plans 2021 to the plan executing unit 1809.

When selection instructions 2014 and 2015 for selecting correspondingdemand plans are input by the user through the input unit 105 (an inputprocess 2002), the selection instructions 2014 and 2015 indicating inputinformation items are respectively transmitted to the plan executingunit 1809 and the smart information superimposing unit 1807.

In the smart information superimposing unit 1807 which has received theselection instruction 2015, a demand plan selecting unit 1901 selectsone of the demand plans included in the presentation information items2007, based on the selection instruction 2015.

The smart information superimposing unit 1807 superimposes thepresentation data item based on the selected demand plan onto thepicture data item 215 with identification information items receivedfrom the appliance identifying unit 104 (a smart superimposing process2016).

The smart information superimposing unit 1807 superimposes presentationdata items on a picture data item 2017 by performing a smartsuperimposing process 2016, and transmits the picture data item 2017with the presentation data items to the output unit 108. The output unit108 outputs the picture data item 2017 with the presentation data itemsusing a liquid crystal display and/or a speaker (an output process2018).

In addition, the plan executing unit 1809 which has received theselection instruction 2014 selects one of the demand plans, based on theselection instruction 2015, and transmits, to an appliance control unit110, a control instruction for causing the corresponding energyconsuming appliance to operate based on the selected demand plan (a planexecuting process 2022).

The appliance control unit 110 transmits the control signal 225corresponding to the received control instruction 223 to each of theenergy consuming appliances, and controls the energy consuming appliance(an appliance control process 224).

Each of FIGS. 21A, 21B, and 21C is a diagram showing an example wherethe image data item 2017 with the superimposed presentation data itemsin this embodiment is output on a liquid crystal display 511 of theoutput unit 108.

The energy consuming appliances displayed on the liquid crystal display511 are the same among FIG. 21A to 21C respectively showing cases wheredifferent demand plans are selected.

FIG. 21A shows a case where the demand plan selecting unit 1901 selectsa demand plan referred to as “Plan 1” based on the selection instruction2015 input through the input unit 105.

In the demand plan “Plan 1” in this case, a cooling temperature set forthe energy consuming appliance (air conditioner) is determined as 28° C.In this case, a current temperature is different from the settemperature, and thus data items such as an image data item 2101 whichvisually enhances the energy consuming appliance (air conditioner) and atext data item 1202 of “Please increase setting temperature to 28° C.!”are superimposed and displayed as presentation data items for the energyconsuming appliance (air conditioner).

FIG. 21B shows a case where the demand plan selecting unit 1901 selectsa demand plan referred to as “Plan 2” based on the selection instruction2015 input through the input unit 105.

In the demand plan “Plan 2” in this case, an illuminance level set forthe energy consuming appliance (lighting equipment) is determined asLevel 5. A current illuminance level is different from the setilluminance level, and thus an image data item 2111 which visuallyenhances the energy consuming appliance (lighting equipment) and a textdata item 2112 of “Please lower illuminance level to Level 5!” aresuperimposed and displayed as presentation data items for the energyconsuming appliance (lighting equipment).

In addition, FIG. 21C shows a case where the demand plan selecting unit1901 selects a demand plan referred to as “Plan 3” based on theselection instruction 2015 input through the input unit 105.

In the demand plan “Plan 3” in this case, a sound volume level set forthe energy consuming appliance (television receiver) is determined asLevel 20. A sound volume level is different from the set sound volumelevel, and thus data items such as an image data item 2121 whichvisually enhances the energy consuming appliance (television receiver)and a text data item 2122 of “Please lower sound volume level to Level20!” are superimposed and displayed as presentation data items for theenergy consuming appliance (television receiver).

In this way, since the differences between the target values in thedemand plan and current values are shown, the user can select the demandplan which satisfies the need, with an increase in the userfriendliness.

In addition, the user can select the demand plan which does not placemuch burden on the user, which increases a possibility that the userexecutes a demand plan for an energy consuming appliance which is notsubject to automatic control, and results in achievement of a demandcontrol request.

Embodiment 7

An energy management system in this embodiment is characterized bygenerating a plurality of demand plans including demand plans in whichappliance configurations are changed by replacement with or addition ofa newly bought energy consuming appliance, and superimposingpresentation data items based on a demand plan selected by a user.

The basic structure of this embodiment is largely in common to that ofEmbodiment 6. Thus, the structural elements used in common to those ofEmbodiment 6 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

FIG. 22 is a diagram showing a configuration of the energy managementsystem in this embodiment.

The energy management system in this embodiment includes an applianceinformation database 2231 which stores specifications information itemsand appliance identification information items on various kinds ofenergy consuming appliances. The appliance information database 2231 isconnected to a management unit 100 through a network 2220.

A demand plan generating unit 2206 in this embodiment generates theplurality of demand plans including a demand plan in which applianceconfigurations are changed by replacement with or addition of a newlybought energy consuming appliance.

The appliance information database 2231 stores the specificationsinformation items and appliance identification information items onenergy consuming appliances 151 to 15 n in user's home and other variouskinds of energy consuming appliances, and further stores component dataitems of the presentation data items.

The appliance information database 2231 stores, as the specificationsinformation items of the energy consuming appliances, information aboutenergy consumed in various kinds of operation states of the energyconsuming appliances and ID of the energy consuming appliances in anassociated manner.

In addition, the appliance information database 2231 stores, as theappliance identification information items of energy consumingappliances, computer graphics data items (CG data items) of the energyconsuming appliances and feature information items of the energyconsuming appliances. In other words, the appliance information database2231 stores CG data items as three-dimensional models of the energyconsuming appliances and the feature information items, of the energyconsuming appliances, for identifying the energy consuming appliances byimage analysis in the picture data items obtained by imaging the energyconsuming appliances. The appliance identification information items ofthe energy consuming appliances are stored together with ID of theenergy consuming appliances in the associated manner.

In addition, the appliance information database 2231 stores text messagetemplates, image data items, audio data items, animation data items etc.together with the ID of the energy consuming appliances in theassociated manner, as the component data items of the presentation dataitems of the energy consuming appliances.

The demand plan generating unit 2206 in this embodiment generates thedemand plans, for example, based on a method performed by a simulationexecuting unit to generate reduction plans described in paragraphs[0078] to [0106] of Patent Literature 2.

The demand plan generating unit 2206 obtains the specificationsinformation items of the various kinds of energy consuming appliancesthrough the network 2220, performs a simulation for a combination of acurrent appliance configuration and each of control patterns, evaluatesthe result of the simulation, derives some of the control patterns whichwill achieve the demand control request (corresponding to the reductiontarget in Patent Literature 2). The details of the derived controlpatterns are regarded as the demand plans.

The demand plan generating unit 2206 generates presentation informationitems including the generated demand plans, and transmits them to thesmart information superimposing unit 2207.

When the operation for selecting one of the demand plans is performed bythe user, the selection instruction indicating the selected demand planis transmitted to the smart information superimposing unit 2207 throughthe input unit 105.

The smart information superimposing unit 2207 selects the demand plancorresponding to the demand plan indicated by the selection instructionreceived through the input unit 105, and superimposes the presentationdata items based on the selected demand plan onto a picture data itemobtained by imaging energy consuming appliances in user's home.

In addition, the demand plan generating unit 2206 transmits, to a planexecuting unit 1809, a demand plan for the same appliance configurationas a current appliance configuration from among the generated demandplans.

The selection instruction indicated by the selected demand plan is alsotransmitted to the plan executing unit 1809 through the input unit 105.The plan executing unit 1809 executes the demand plan indicated by theselection instruction received through the input unit 105.

FIG. 23 shows an internal structure of the smart informationsuperimposing unit 2207 in this embodiment.

The smart information superimposing unit 2207 in this embodiment issimilar to the smart information superimposing unit 1807 in Embodiment 6except that a presentation data generating unit 2302 operatesdifferently from the presentation data generating unit 302 in Embodiment6.

The presentation data generating unit 2302 performs operations indicatedbelow when an energy consuming appliance determined by the presentationneed determining unit 301 as being an energy consuming appliance whichneeds to be presented is different from any of the energy consumingappliances included in the current appliance configuration, in otherwords, performs operations for a demand plan showing that a new energyconsuming appliance is added simply or as a replacement therein. In thiscase, the presentation data generating unit 2302 obtains the applianceidentification information item of the energy consuming appliance whichis not included in the current appliance configuration from theappliance information database 2231 through the network 2220.

The presentation data generating unit 2302 generates a presentation dataitem of the energy consuming appliance using the applianceidentification information item obtained from the appliance informationdatabase 2231.

For example, the presentation data generating unit 2302 generates animage data item presenting a state in which a replacement energyconsuming appliance is disposed, using a CG data item of the energyconsuming appliance included in the appliance identification informationitem, together with a message for proposing a buy of the replacementenergy consuming appliance.

When the energy consuming appliance determined by the presentation needdetermining unit 301 as being an energy consuming appliance which needsto be presented is the same as one of the energy consuming appliancesincluded in the current appliance configuration, the presentation datagenerating unit 2302 generates presentation data items including imagedata items, audio data items, text data items, video data items, etc. asperformed by the presentation data generating unit 302. In this case,when a rate ((Er−Ep)/Ep) of the difference between Er and Ep withrespect to the demand plan is larger than or equal to a predeterminedrate, the presentation data generating unit 2302 generates, as apresentation data item, a piece of advice for prompting the user toperform the control specified by the demand plan.

In addition, when a rate ((Er−Ep)/Ep) of the difference between Er andEp with respect to the demand plan is smaller than the predeterminedrate, the presentation data generating unit 2302 generates, as apresentation data item, an information item indicating a currentoperation state, and an information item indicating that the currentoperation state is almost ideal for achieving the demand controlrequest.

FIG. 24 is a diagram showing a sequence of operations performed by theenergy management system in this embodiment.

The following sequences are similar to those in Embodiment 6: a sequenceof processes from an appliance observing process 201 by the applianceobserving unit 101 to when appliance specification information items 205are transmitted to the demand plan generating unit 1806; and a sequencefrom an image capturing process 211 by the camera unit 102 and to when apicture data item 215 with the identification information items istransmitted to the smart information superimposing unit 107.

In this embodiment, the demand plan generating unit 2206 performs ademand plan generating process 2406. In this process, the demand plangenerating unit 2206 obtains specifications information items 2405 ofthe various kinds of energy consuming appliances from the applianceinformation database 2231, in addition to the appliance specificationsinformation 205. In this way, the demand plan generating unit 2206generates the plurality of demand plans including the demand plans inwhich appliance configurations are changed by replacement with oraddition of a newly bought energy consuming appliance.

The demand plan generating unit 2206 generates presentation informationitems 2407 including the generated demand plans, and transmits them tothe smart information superimposing unit 2207.

When a selection instruction 2415 for selecting a corresponding demandplan is input by the user through the input unit 105 (an input process2402), the selection instruction 2415 indicating an input informationitem is transmitted to the smart information superimposing unit 2207.

In the smart information superimposing unit 2207 which has received theselection instruction 2415, a demand plan selecting unit 1901 selectsone of the demand plans included in the presentation information items2407, based on the selection instruction 2415.

Furthermore, in the smart information superimposing unit 2207, thepresentation data generating unit 2302 obtains, as necessary, anappliance identification information item of an energy consumingappliance which is not included in a current appliance configuration,from the appliance information database 2231, and generates thepresentation data item using also the appliance identificationinformation item.

Furthermore, the smart information superimposing unit 2207 superimposesthe presentation data item generated by the superimposing unit 303 ontothe picture data item 215 with the identification information itemreceived from the appliance identifying unit 104 (a smart superimposingprocess 2416).

The smart information superimposing unit 2207 performs the smartsuperimposing process 2416, and transmits the picture data item 2417with the superimposed presentation data item to the output unit 108.

The output unit 108 outputs the image data item 2417 with thesuperimposed presentation data item using the liquid crystal displayand/or the speaker (an output process 2418).

In addition, the demand plan generating unit 2206 transmits thepresentation information items 2407 showing the generated demand plansto the smart information superimposing unit 2207, and transmits, to theplan executing unit 1809, a demand plan 2421 for the same applianceconfiguration as a current appliance configuration from among thegenerated demand plans.

When a selection instruction 2414 for selecting a corresponding demandplan is input by the user through the input unit 105 (an input process2402), the selection instruction 2414 indicating an input informationitem is transmitted to the plan executing unit 1809.

In addition, the plan executing unit 1809 which has received theselection instruction 2414 selects one of the demand plans, based on theselection instruction 2414, and transmits, to an appliance control unit110, a control instruction for causing the corresponding energyconsuming appliance to operate based on the selected demand plan (a planexecuting process 2022).

The appliance control unit 110 transmits the control signal 225corresponding to the received control instruction 223 to each of theenergy consuming appliances, and controls the energy consuming appliance(an appliance control process 224).

Each of FIGS. 25A, 25B, and 25C is a diagram showing an example wherethe image data item 2417 with the superimposed presentation data item inthis embodiment is output on the liquid crystal display 511 of theoutput unit 108.

FIGS. 25A, 25B, and 25C respectively show cases in which mutuallydifferent demand plans are selected.

FIG. 25A shows a case where the demand plan selecting unit 1901 selectsa demand plan referred to as “Plan 1” which is a demand plan without anychange in the appliance configuration, based on the selectioninstruction 2415 input through the input unit 105.

In the demand plan “Plan 1” in this case, a cooling temperature set forthe energy consuming appliance (air conditioner) is determined as 28° C.which is different from a current temperature. Furthermore, as for theenergy consuming appliance (air conditioner), an image data item 2501which visually enhances the energy consuming appliance (air conditioner)and a text data item 2502 of “Please increase setting temperature to 28°C.!” are superimposed and displayed as presentation data items.

In addition, FIG. 25B shows a case where the demand plan selecting unit1901 selects a demand plan referred to as “Plan 2” without a change inthe appliance configuration, based on the selection instruction 2415input through the input unit 105.

In the demand plan “Plan 2” in this case, the energy consuming appliance(air conditioner) is scheduled to be replaced with a new one. For thisreason, as presentation data items of the energy consuming appliance(air conditioner), the following data items are superimposed anddisplayed: an image data item 2510 showing a state in which areplacement energy consuming appliance is disposed; an image data item2511 which visually enhances the energy consuming appliance (airconditioner); and a text data item 2512 of “Please buy air conditionerwith product number X!” for prompting the user to buy the new one.

In addition, FIG. 25C shows a case where the demand plan selecting unit1901 selects a demand plan referred to as “Plan 3” with a change in theappliance configuration, based on the selection instruction 2015 inputthrough the input unit 105.

In the demand plan “Plan 3” in this case, the energy consuming appliance(heat exchanger) is scheduled to be replaced with a new one. For thisreason, as presentation data items of the energy consuming appliance(heat exchanger), the following data items are superimposed anddisplayed: an image data item 2520 showing a state in which areplacement energy consuming appliance (heat exchanger) is disposed; animage data item 2521 which visually enhances the energy consumingappliance (heat exchanger); and a text data item 2522 of “Please buyheat exchanger with product number Y!” for prompting the user to buy thenew one.

In this way, the demand plans in which appliance configurations arechanged by replacement with or addition of the newly bought energyconsuming appliance are presented, which enables the user to select ademand plan for satisfying the need from among an increased number ofoptions. This results in increase in the userfriendliness.

In addition, the user can select the demand plan which does not placemuch burden on the user, which increases a possibility that the userexecutes a demand plan for an energy consuming appliance which is notsubject to automatic control and results in achievement of a demandcontrol request.

Here, when the user buys a new energy consuming appliance forreplacement or addition, the information stored by the applianceinformation storing unit 103 is updated.

When the new energy consuming appliance for replacement or addition isbought, the appliance information storing unit 103 obtains aspecifications information item and an appliance identificationinformation item of the new energy consuming appliance through thenetwork 2220, and updates the information stored in the applianceinformation storing unit 103.

In addition, when the new energy consuming appliance for replacement oraddition is bought, the component data of the presentation data storedby the presentation data generating unit 2302 is also updated.

When the new energy consuming appliance for replacement or addition isbought, the presentation data generating unit 2302 obtains the componentdata item of the presentation data item of the new energy consumingappliance through the network 2220, and updates the component data ofthe presentation data stored by the presentation data generating unit2302.

Here, in the selected demand plan, the presentation need determiningunit 301 may determine the presentation data item related to the newenergy consuming appliance to be a presentation data item which needs tobe presented preferentially.

FIG. 26 is a flowchart of the presentation need determining processperformed by the presentation need determining unit 301 in this case.

First, based on appliance identification information items of theappliances, the presentation need determining unit 301 counts (i) thenumber N (N is an integer larger than or equal to 0) of energy consumingappliances identified in the picture data item 212, and (ii) the numberNc (Nc is an integer larger than or equal to 0) of energy consumingappliances subject to change in the selected demand plan (Step S3401).

Next, the presentation need determining unit 301 quantifies thedifferences between target values obtainable by performing the demandplan and current values related to the identified N energy consumingappliances (Step S402). Examples of data specifically used here includecontrol details for the energy consuming appliances indicated by thedemand plan. In the case where the demand control request is a controlrequest related to a total energy consumption amount, the difference ΔE(=Er−Ep) between an estimated value Ep and an actual value Er iscalculated as being an obtainable value when the control details areperformed, wherein Ep denotes the amount of energy consumed by eachenergy consuming appliance and Er denotes the amount of energy consumedunder a current state.

In the case where the demand control request is a control requestrelated to a CO2 discharge amount, the difference ΔE (=Er−Ep) between anestimated value Ep and an actual value Er is calculated as beingobtainable value when the control details are performed, wherein Epdenotes the CO2 discharge amount based on energy consumed by each energyconsuming appliance and Er denotes the CO2 discharge amount based onenergy consumed under a current state.

In the case where the demand control request is a control requestrelated to cost, the difference ΔE (=Er−Ep) between an estimated valueEp and an actual value Er is calculated as being obtainable value whenthe control details are performed, wherein Ep denotes cost based onenergy consumed by each energy consuming appliance and Er denotes costbased on energy consumed under a current state.

Next, the number N of energy consuming appliances is compared with themaximum presentation number M (M is an integer larger than or equalto 1) indicating the number of presentation data items to be presentedat maximum (Step S403).

When N≤M is satisfied, the number of energy consuming appliancesidentified in the picture data item 212 is smaller than or equal to themaximum presentation number M. Thus, presentation data items of N energyconsuming appliances among the identified appliances identified in thepicture data item are determined to be presentation data items whichneed to be presented, for completion of the presentation needdetermining process (Step S405).

When N>M is satisfied, the number Nc of energy consuming appliances inthe selected plan is also compared with the maximum presentation numberM for the presentation data items (Step S3403).

When Nc≥Md is satisfied, M energy consuming appliances are selected fromamong the appliances subject to change in the selected plan indescending order of differences ΔE between target values in the demandplan and current values, and are determined to be energy consumingappliances for which presentation data items need to be presented, forcompletion of the presentation need determining process (Step S3404).

In addition, when Nc<M, Nc energy consuming appliances in the selectedplan are determined to be energy consuming appliances for whichpresentation data items need to be presented (Step S3405). Furthermore,M−Nc energy consuming appliances are selected from among the remainingN−Nc energy consuming appliances in descending order of differences ΔEbetween target values in the demand plan and current values calculatedin Step S402. The presentation data items of M−Nc energy consumingappliances are determined to be presentation data items which need to bepresented, for completion of the presentation need determining process(Step S3406).

In this case, when the selected demand plan shows that one of the energyconsuming appliances is changed, the message for prompting the user tobuy the new energy consuming appliance for replacement is preferentiallypresented to the user.

Embodiment 8

An energy management system in this embodiment is characterized in thatenergy consuming appliances operate in a new operation mode, and newpresentation data items are further superimposed.

In this embodiment, the following data items are obtained. Depending oncombinations of low-level commands to the energy consuming appliances,control modules for causing the energy consuming appliances to operatein the new operation mode, data items including the new presentationdata items (hereinafter referred to as “appliance content items”)corresponding to the new operation mode are obtained through a network.This allows the energy consuming appliances to operate in the newoperation mode, and enables superimposition of the new presentation dataitems.

Here, the new operation mode is an operation mode which cannot benormally used by energy consuming appliances, and is, for example, anenergy saving control mode based on a new algorithm.

In addition, the new presentation data items are used to present, to theuser, messages corresponding to the new operation mode and image dataitems, and are not output by simply connecting the energy consumingappliances to the management unit 100.

The basic structure of this embodiment is largely in common to that ofEmbodiment 3. Thus, the structural elements used in common to those ofEmbodiment 3 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

FIG. 27 is a diagram showing a configuration of the energy managementsystem in this embodiment.

The energy management system in this embodiment further includes anappliance content server 2631 which distributes the appliance contentitems for the various kinds of energy consuming appliances. Theappliance content server 2631 is connected to the management unit 100through the network 2220.

Furthermore, the management unit 100 includes an appliance contentmanaging unit 2630 which manages the appliance content items downloadedfrom the appliance content server 2631.

FIG. 28 shows an internal structure of the appliance content managingunit 2630.

The appliance content managing unit 2630 includes an appliance contentstoring unit 2701, an appliance content obtaining unit 2702, and anappliance content selecting unit 2703.

The appliance content storing unit 2701 stores the content items 2711 to271 k (k is an integer larger than or equal to 1).

The appliance content obtaining unit 2702 obtains the appliance contentitems from the appliance content server 2631 through the network 2220,and stores them in the appliance content storing unit 2701.

The appliance content selecting unit 2703 selects appliance contentitems to be applied to the energy consuming appliances 151 to 15 n fromamong appliance content items stored in the appliance content storingunit 2701.

The appliance content items include: a control module 2721 for causingthe energy consuming appliances to operate in the new operation mode;and specifications information items 2722 of the energy consumingappliances and presentation items 2723 which are used to generatepresentation data items for the energy consuming appliances to becontrolled by the control module 2721.

The control module 2721 is a software program which is, for example,obtained by combining low-level commands to the energy consumingappliances, and is used to cause the energy consuming appliances tooperate the new operation mode.

The specifications information items 2722 are information items relatedto energy consumption in the new operation mode in the case where theenergy consuming appliances are controlled by the control module 2721.

The presentation items 2723 are component data items of the newpresentation data items corresponding to the new operation mode, andinclude text message templates, image data items, audio data items,animation data items, etc.

The appliance content obtaining unit 2702 obtains the appliance contentitems from the appliance content server 2631. This obtainment isperformed based on a user instruction which is input through an inputunit 105. In addition, this obtainment is performed by periodicallyaccessing the appliance content server 2631, and based on setting forobtaining obtainable appliance content items. In addition, thisobtainment is performed in response to reception of event informationshowing that a new energy consuming appliance is newly connected to themanagement unit 100.

Here, in the obtainment of the appliance content item from the appliancecontent server 2631, a buy and sell agreement for the appliance contentitem or a use agreement for the appliance content item may beauthenticated as necessary.

In addition, it is desirable that the appliance content be transmitted,for example, after being encrypted for the purpose of preventingunauthorized obtainment by a third person.

The appliance content storing unit 2701 stores the appliance contentitems and ID of the energy consuming appliances in an associated manner.

For each of the energy consuming appliances, plural appliance contentitems may be stored therein, or no appliance content item may be storedtherein.

The appliance content selecting unit 2703 selects appliance contentitems suitable for the energy consuming appliances 151 to 15 n. Thisselection is performed based on one of the following: a user instructionwhich is input through the input unit 105; a demand plan which isgenerated by the demand plan generating unit 2606; a use agreementinformation of the appliance content; and event information indicatingthat a new energy consuming appliance is newly connected to themanagement unit 100.

In relation to the selected appliance content, the appliance contentselecting unit 2703 transmits (i) the control module 2721 to a planexecuting unit 2609, (ii) the specifications information item 2722 tothe demand plan generating unit 2606; and the presentation item 2723 toa smart information superimposing unit 2607.

Here, when the appliance content item is selected based on the useagreement information, for example, it is desirable that the appliancecontent item be encrypted and stored in the appliance content storingunit 2701 so as to prevent the appliance content item from beingselected in an unauthorized manner.

For example, a case is conceivable in which the appliance contentselecting unit 2703 selects an appliance content item 271 j (j is aninteger 1≤j≤k) for an energy consuming appliance 15 i (j is an integer1≤i≤n). In this case in this embodiment: (i) the demand plan generatingunit 2606 generates, for the energy consuming appliance 15 i, a demandplan with reference to the specifications information item 2722 of theappliance content item 271 j; (ii) the plan executing unit 2609 controlsthe energy consuming appliance 15 i using the control module of theappliance content item 271 j; and (iii) the smart informationsuperimposing unit 2607 generates the presentation data item of theenergy consuming appliance 15 i using the presentation item 2723 of theappliance content item 271 j.

FIG. 29 is a diagram showing a sequence of operations performed by theenergy management system in this embodiment.

The following sequences are similar to those in Embodiment 3: a sequenceof processes from the appliance observing process 201 by the applianceobserving unit 101 and the input process 202 for a demand controlrequest through the input unit 105 to when appliance specificationsinformation items 205 are transmitted to the demand plan generating unit2606; and a sequence from an image capturing process 211 by a cameraunit 102 and to when the picture data item 215 with the identificationinformation items is transmitted to the smart information superimposingunit 107.

In the energy management system in this embodiment, the appliancecontent obtaining unit 2702 of the appliance content managing unit 2630receives the appliance content item 2801 from the appliance contentserver 2631, and stores the appliance content storing unit 2701 (anobtaining process 2802).

When a selection instruction 2803 for the appliance content item isinput through the input unit 105 (an input process 2802 i), the inputselection instruction 2803 is transmitted to the appliance contentmanaging unit 2630.

In the appliance content managing unit 2630, the appliance contentselecting unit 2703 (i) selects the appliance content item based on thereceived selection instruction 2803, (ii) transmits a control module2804 of the selected appliance content item to the plan executing unit2609, (iii) transmits a specifications information item 2805 to thedemand plan generating unit 2606, and (iv) transmits the presentationitem 2806 to the smart information superimposing unit 2607.

The demand plan generating unit 106 generates a demand plan whichsatisfies a demand control request 203 with reference to (i) thespecifications information item 2805 received from the appliance contentmanaging unit 2630 as for the energy consuming appliance for which theappliance content item is applied, (ii) appliance specificationsinformation item 205 stored in the appliance information storing unit103 as for the energy consuming appliance for which the appliancecontent item is not applied, (iii) appliance operation state informationitem 204 stored in a memory area which is obtainable by accessing theappliance observing unit 101 (a demand plan generating process 2806 p).

The demand plan generating unit 2606 transmits the presentationinformation item 2807 including the generated demand plan to the smartinformation superimposing unit 2607, and transmits the generated demandplan 2821 to the plan executing unit 2609.

Upon receiving the demand plan 2821, the plan executing unit 2609transmits the control instruction 2823 to the appliance control unit2610, so that the energy consuming appliance is caused to operate basedon the details of the demand plan 2821, using the control module 2804received from the appliance content managing unit 2630 (a plan executingprocess 2822).

The appliance control unit 2610 transmits the control signal 2825corresponding to the received control instruction 2823 to each of theenergy consuming appliances, and controls the energy consuming appliance(an appliance control process 2824).

Upon receiving the presentation information item 2807 and the picturedata item 215 with identification information items, the smartinformation superimposing unit 2607 generates presentation data itemsusing the presentation item 2806 received from the appliance contentmanaging unit 2630, and superimposes the generated presentation dataitems around the energy consuming appliances on the picture data item215 (a smart superimposing process 2816).

The smart information superimposing unit 2607 transmits the picture dataitem 2817 with the superimposed presentation data items. The output unitoutputs the picture data item 2817 with the superimposed presentationdata items on a liquid crystal display and/or using a speaker (an outputprocess 2818).

Here, the demand plan generating unit 106 performs a simulation on acombination of appliance content items for the energy consumingappliances 151 to 15 n, and may generate the optimum demand plan whichsatisfies the demand control request. In this case, the demand plangenerating unit 2606 performs the simulation after receiving thespecifications information items of the appliance content items otherthan an appliance content item which is currently being selected, fromthe appliance content selecting unit 2703.

In this case, as a result, the demand plan may include applying anappliance content item which is different from the current appliancecontent item to the energy consuming appliance.

Each of FIGS. 30A, 30B, and 30C is a diagram showing an example wherethe picture data item 2817 with the superimposed presentation data itemin this embodiment is output on the liquid crystal display 511 of theoutput unit 108.

FIGS. 30A, 30B, and 30C respectively show cases where differentappliance content items are selected.

FIG. 30A shows and an example case where an appliance content item isused for an energy consuming appliance (air conditioner).

In this case, the appliance content item used for the energy consumingappliance (air conditioner) includes a control module for operation in anew energy saving mode referred to as “comfort energy saving mode”.

On the liquid crystal display 511, a display application displays andsuperimposes, as presentation data items for the energy consumingappliance (air conditioner), the following presentation data items: animage data item 2901 which visually enhances the energy consumingappliance (air conditioner) in a format different from a normal format;a text data item 2902 of “Operating in comfort energy saving mode!”; anda unique icon 2903.

In addition, when the user taps the icon 2903 on a touch panelintegrated with the liquid crystal display 511, a speech of “Operatingin comfort energy saving mode!” is output by an audio reproductionapplication from the speaker 512.

FIG. 30B shows an example case where an appliance content item is usedfor an energy consuming appliance (television receiver).

The appliance content item showing the energy consuming appliance(television receiver) includes a control module for operation in a newenergy saving mode designed for children which is referred to as “energysaving mode for children”.

For example, this control module automatically turns OFF a power sourceof the energy consuming appliance (television receiver) when the ONstate of the power source exceeds a predetermined period of time, andlowers a sound volume level of the energy consuming appliance(television receiver) to a level at or below a certain level.

On the liquid crystal display 511, the display application displays andsuperimposes, as presentation data items for the energy consumingappliance (television receiver), the following presentation data items:an image data item 2911 which visually enhances the energy consumingappliance (television receiver) in a format different from a normalformat; a text data item 2912 of “Please let television receiver sleepsometimes!”; and a unique icon for children 2913.

In addition, when the user taps the icon 2913 on the touch panelintegrated with the liquid crystal display 511, a speech of “Please lettelevision receiver sleep sometimes!” is output by the audioreproduction application from the speaker 512.

FIG. 30C shows an example case where an appliance content item showingan energy consuming appliance (lighting equipment) is used and anappliance content item for the “comfort energy saving mode” is requiredfor the energy consuming appliance (air conditioner) in order to satisfythe demand control request.

In this case, the appliance content item used for the energy consumingappliance (lighting equipment) includes a control module for operationsin new energy saving modes collectively referred to as “energy savingmodes for time zones”.

This control module changes illuminance levels of the light dependentlyon time zones such as morning, noon, evening, and night.

On the liquid crystal display 511, the display application displays andsuperimposes, as presentation data items for the energy consumingappliance (lighting equipment), the following presentation data items:an image data item 2921 which visually enhances the energy consumingappliance (lighting equipment) in a format different from a normalformat; a text data item 2922 of “Illuminance level 3 in current timezone!”; and a unique icon 2923.

Furthermore, data items such as an image data item 501 which visuallyenhances the energy consuming appliance (air conditioner) and a textdata item 2924 of “Appliance content for comfortable energy saving moderecommended!” are superimposed and displayed as presentation data itemsfor the energy consuming appliance (air conditioner).

In addition, when the user taps the icon 2923 on the touch panelintegrated with the liquid crystal display 511, a speech of “Illuminancelevel 3 in current time zone!” is output by the audio reproductionapplication from the speaker 512.

In addition, when the user taps the energy consuming appliance (airconditioner), a speech of “Appliance content for comfortable energysaving mode recommended!” is output by the audio reproductionapplication from the speaker 512.

The use of the appliance content items for the energy consumingappliances allows the energy consuming appliances to operate in newoperation modes such as new energy saving modes, which increases thepossibility that the demand control request can be achieved.

Furthermore, since the new presentation data items corresponding to thenew operation modes are superimposed and presented, it is possible topresent, to the user, operation states of the energy consumingappliances in the new operation modes even when the display on the LED,liquid crystal display, or the like is unchanged.

Here, it is also good to include, as presentation items, image or videodata items and audio data items related to celebrities such as movieactors and actresses and characters in animation works.

In this case, since the image or video data items and audio data itemsrelated to the celebrities and characters are superimposed and presentedto the user, the presentation data items are more likely to draw user'sattention than normal presentation data items, with an increasedvisibility.

Embodiment 9

An energy management system in this embodiment is characterized in thatpresentation data items are presented to a user in synchronization withoperations by energy consuming appliances and that the energy consumingappliances operate in synchronization with presentation of thepresentation data items to the user.

In some cases, timing at which a presentation data item for an energyconsuming appliance is presented to a user with a delay from timing ofan operation by the energy consuming appliance. In such a case, the usermay think the presentation data item strange and ignore the presentationdata item.

When the presentation data item is presented to the user insynchronization with the operation by the energy consuming appliance,the user does not think the presentation data item strange and paysattention to the presentation data item, resulting in an increase in thevisibility.

The basic structure of this embodiment is largely in common to that ofEmbodiment 8. Thus, the structural elements used in common to those ofEmbodiment 8 are assigned with the same numerical references, andindication of diagrams and detailed descriptions thereof are omittedwhere appropriate.

FIG. 31 is a diagram showing a system configuration of an energymanagement system in this embodiment.

FIG. 32 shows an internal structure of the appliance content managingunit 3030 in this embodiment.

The energy management system in this embodiment is similar to the one inEmbodiment 8, except for an appliance content managing unit 3030, anoutput unit 3008, and an appliance control unit 3010.

In this embodiment, the appliance content managing unit 3030 furtherincludes the output unit 3008, and a synchronous control unit 3105 whichperforms synchronous control on the appliance control unit 3010.

The appliance content managing unit 3030 communicates synchronouscontrol information items with each of the output unit 3008 and theappliance control unit 3010. In this way, the appliance content managingunit 3030 helps the output unit 3008 to output the presentation dataitems in synchronization with operations by the energy consumingappliances. In addition, the appliance content managing unit 3030 helpsthe appliance control unit 3010 to control the energy consumingappliances in synchronization with presentation of the presentation dataitems to the user.

The synchronous control information items include ID for identifying theenergy consuming appliances, ID indicating operations to besynchronized, and time information items about time points at which theoperations are to be executed. The time information items included inthe synchronous control information items are classified into timeinformation items based on absolute time Ta (Ta shows a time pointrepresented by “specific time, minute, and second”) and time informationitems based on relative time Tr (Tr shows a time point represented as,for example, “after N seconds”).

For example, the output unit 3008 receives, from the synchronous controlunit 3105, a synchronous control information item including ID foridentifying one of the energy consuming appliances, an identificationinformation item indicating a reproduction operation of an animationitem, and an absolute time Ta. In response to the reception of thesynchronous control information item, the output unit 3008 performs anoperation of outputting the data items including the superimposedanimation item as the presentation data items of the energy consumingappliance at the absolute time Ta.

As another example, the output unit 3008 receives, from the synchronouscontrol unit 3105, a synchronous control information item including IDfor identifying one of the energy consuming appliances, anidentification information item indicating a reproduction operation ofan animation work, and an absolute time Tr. In response to the receptionof the synchronous control information item, the output unit 3008performs an operation of outputting the data items including thesuperimposed animation item as the presentation data items of the energyconsuming appliance after the elapse of the relative time Tr.

For example, the appliance control unit 3010 receives, from thesynchronous control unit 3105, a synchronous control information itemincluding ID for identifying one of the energy consuming appliances, anidentification information item indicating a certain operation by theenergy consuming appliance, and an absolute time Ta. In response to thereception of the synchronous control information item, the appliancecontrol unit 3010 causes the energy consuming appliance to perform theoperation at the absolute time Ta.

As another example, the appliance control unit 3010 receives, from thesynchronous control unit 3105, a synchronous control information itemincluding ID for identifying one of the energy consuming appliances, anidentification information item indicating a certain operation by theenergy consuming appliance, and an absolute time Tr. In response to thereception of the synchronous control information item, the appliancecontrol unit 3010 causes the energy consuming appliance to perform theoperation at the relative time Tr.

FIG. 33 is a diagram showing a sequence of operations performed by anenergy management system in this embodiment.

The following sequence of processes is similar to Embodiment 8. Thesequence similar to the one in Embodiment 8 includes: an applianceobserving process 201 by an appliance observing unit 101; and processesfrom an obtaining process 2802 for obtaining an appliance content item2801 performed by an appliance content managing unit 3030 to when acontrol module 2804, a specifications information item 2805, and apresentation item 2806 each for the selected appliance content item aretransmitted to a plan executing unit 2609, a demand plan generating unit2606, and a smart information superimposing unit 2607, respectively. Inaddition, the following sequences are also similar to those inEmbodiment 8: a sequence from an input process 202 for a demand controlrequest to the input unit 105 to when a control instruction 2823 istransmitted to the appliance control unit 3010; and a sequence from animage capturing process 211 by a camera unit 102 to when a picture dataitem 2817 with a presentation data item superimposed thereon.

In the energy management system in this embodiment, the synchronouscontrol unit 3105 of the appliance content managing unit 3030 performs asynchronous control process 3200, communicates a synchronous controlinformation item 3201 with the output unit 3008, and communicates asynchronous control item 3202 with the appliance control unit 3010.

Based on the received synchronous control information item 3201, theoutput unit 3008 outputs a picture data item 2817 with superimposedpresentation data items, using a liquid crystal display and/or a speaker(an output process 3218). The appliance control unit 3010 controls theenergy consuming appliances based on the synchronous control informationitem 3202 (an appliance control process 3224).

Each of FIGS. 34A and 34B is a schematic diagram showing an examplestate where operations by the energy consuming appliances andpresentation of presentation data items to the user are performed insynchronization with each other in this embodiment.

FIG. 34A shows an example state where a presentation data item for theenergy consuming appliance (air conditioner) 152 is output onto theoutput unit 3008 in synchronization with the operation by the energyconsuming appliance 152 (air conditioner).

In this case, as known from an appliance content item, the energyconsuming appliance (air conditioner) 152 operates in a new energysaving mode referred to as “Energy saving mode for health”.

On a liquid crystal display 511, an image data item is displayed whichis obtained by imaging the energy consuming appliance (air conditioner)152 in user's home and with a superimposed presentation data itemrelated to the energy consuming appliance (air conditioner) 152.

Presentation data items superimposed in this case are nanoparticle ions3301 having an elimination effect to be output from the energy consumingappliance (air conditioner) 152, a unique icon 3303 showing that the“Energy saving mode for health” is currently being performed, and ananimation item showing a flow of wind 3302 which is currently outputfrom the energy consuming appliance (air conditioner) 152.

Based on the synchronous control information received from thesynchronous control unit 3105, the output unit 3008 outputs the imagedata item with the superimposed presentation data item. The image dataitem with the superimposed presentation data item is the image data itemobtained by superimposing the presentation data item onto the image dataitem obtained by imaging the energy consuming appliance (airconditioner) 152 in the home. When the image data item is output basedon the synchronous control information, the animation item is reproducedin synchronization with the flow of wind from the actual energyconsuming appliance (air conditioner) 152.

The nanoparticle ions 3301 and flow of wind 3302 output from the energyconsuming appliance (air conditioner) 152 cannot be seen actually, butare visualized using the presentation data items in this case.

In this way, by presenting, to the user, the presentation data items insynchronization with the operation by the actual energy consumingappliance (air conditioner) 152, it is possible to present the state ofoperation by the energy consuming appliance (air conditioner) 152without causing the user to feel the presentation strange. As a result,the presentation data item attracts the user's attention, resulting inan increase in the visibility.

FIG. 34B shows an example state where the energy consuming appliance(air conditioner) 152 performs a predetermined operation insynchronization with output of an image data item with a superimposedpresentation data item onto the output unit.

In this case, as known from an appliance content item, the energyconsuming appliance (air conditioner) 152 operates in a new energysaving mode referred to as “Comfort saving mode”.

On the liquid crystal display 511, an image data item is displayed whichis obtained by imaging the energy consuming appliance (air conditioner)152 in the home and with the superimposed presentation data item relatedto the energy consuming appliance (air conditioner) 152.

On the liquid crystal display 152, the following presentation data itemsfor the energy consuming appliance (air conditioner) 152 aresuperimposed and displayed as presentation data items: an image dataitem 2901 which visually enhances the energy consuming appliance (airconditioner) 152 in a format different from a normal format; a text dataitem 2902 of “Operating in comfort energy saving mode!”; and a uniqueicon 2903.

In addition, the user taps the icon 2903 on the touch panel integratedwith the liquid crystal display 511. In response to this, the audioreproduction application causes the speaker 512 to output a speech of“Operating in comfort energy saving mode!”, and causes the energyconsuming appliance (air conditioner) 152 to flicker its light emittingdiode (LED) 3301 at the same time.

At this time, based on the synchronous control information received fromthe synchronous control unit 3105, the appliance control unit 3010transmits a control signal for flickering the LED 3301 to the energyconsuming appliance (air conditioner) 152. With this, the energyconsuming appliance (air conditioner) 152 flickers its LED 3301 insynchronization with the output of the speech from the speaker 512.

By flickering the LED 3301 in synchronization with the output of thespeech, it is possible to cause the user to feel as if the energyconsuming appliance (air conditioner) 152 is saying the speech.

In this way, by means of the actual energy consuming appliance operatingin synchronization with the presentation of the presentation data itemto the user, it is possible to present the details of the presentationdata item without causing the user to feel the presentation strange. Asa result, the presentation data item attracts the user's attention,resulting in an increase in the visibility.

[Other Variations]

The present invention has been described above based on the embodiments,but naturally is not limited to the exemplary embodiments. The presentinvention also includes the following implementations.

(1) Each of the aforementioned apparatuses is, specifically, a computersystem including a microprocessor, a ROM, a RAM, a hard disk unit, adisplay unit, a keyboard, a mouse, and so on. A computer program isstored in the RAM or hard disk unit. The respective apparatuses achievetheir functions through the microprocessor's operations according to thecomputer program. Here, the computer program is configured to achievetheir functions by combining plural instruction codes indicatinginstructions for the computer. Computer systems here are not limited tocomputer systems including all of a microprocessor, a ROM, a RAM, a harddisk unit, a display unit, a keyboard, a mouse, and so on, and computersystems including some of the apparatuses are also possible.

(2) A part or all of the structural elements of the respectiveapparatuses may be configured with a single system-LSI (Large-ScaleIntegration). The system-LSI is a super-multi-function LSI manufacturedby integrating structural units on a single chip, and is specifically acomputer system configured to include a microprocessor, a ROM, a RAM,and so on. A computer program is stored in the RAM. The system-LSIachieves its function through the microprocessor's operations accordingto the computer program.

Furthermore, each of the structural units of the respective apparatusesmay be made as separate individual chips, or as a single chip to includea part or all thereof.

Furthermore, system-LSI is mentioned here, but there are instances wherethe designations IC, LSI, super LSI, or ultra LSI are used depending onthe degree of integration. Furthermore, the means for circuitintegration is not limited to an LSI, and implementation with adedicated circuit or a general-purpose processor is also available. Inaddition, it is also possible to use a Field Programmable Gate Array(FPGA) that is programmable after the LSI is manufactured, and areconfigurable processor in which connections and settings of circuitcells within the LSI are reconfigurable.

Furthermore, when a circuit integration technology for replacing LSIswith new circuits appears in the future with advancement insemiconductor technology and derivative other technologies, the circuitintegration technology may be naturally used to integrate functionalblocks. Application of biotechnology is one such possibility.

(3) A part or all of the constituent elements constituting therespective apparatuses may be configured as an IC card which can beattached to and detached from the respective apparatuses or as astand-alone module. The IC card or the module is a computer systemconfigured from a microprocessor, a ROM, a RAM, and so on. The IC cardor the module may also be included in the aforementionedsuper-multi-function LSI. The IC card or the module achieves itsfunctions through the microprocessor's operations according to thecomputer program. The IC card or the module may also be implemented tobe tamper-resistant.

(4) The present invention may be the above-described method.Furthermore, the present invention may be implemented as computerprograms for executing the above-described method, using a computer, andmay also be implemented as digital signals including the computerprograms.

Furthermore, the present invention may also be implemented as computerprograms or digital signals recorded on computer-readable recordingmedia such as a flexible disc, a hard disk, a CD-ROM, an MO, a DVD, aDVD-ROM, a DVD-RAM, a BD (Blu-ray Disc), and a semiconductor memory.Furthermore, the present invention may also be implemented as thedigital signals recorded on these recording media.

Furthermore, the present invention may also be implemented as theaforementioned computer programs or digital signals transmitted via atelecommunication line, a wireless or wired communication line, anetwork represented by the Internet, a data broadcast, and so on.

The present invention may also be implemented as a computer systemincluding a microprocessor and a memory, in which the memory stores theaforementioned computer program and the microprocessor operatesaccording to the computer program.

Furthermore, it is also possible to execute another independent computersystem by transmitting the programs or the digital signals recorded onthe aforementioned recording media, or by transmitting the programs ordigital signals via the aforementioned network and the like.

(5) The embodiments and variations may be arbitrarily combined.

Each of the structural elements in each of the above-describedembodiments may be configured in the form of an exclusive hardwareproduct, or may be realized by executing a software program suitable forthe structural element. Each of the structural elements may be realizedby means of a program executing unit, such as a CPU and a processor,reading and executing the software program recorded on a recordingmedium such as a hard disk or a semiconductor memory. Here, the softwareprogram for realizing the control information display apparatusaccording to each of the embodiments is a program described below.

For example, this program causes a computer to execute a controlinformation display method including: obtaining one or more controlinformation items (i) regarding one or more appliances which consumeenergy provided in a building and (ii) used for satisfying a controlrequest in the building; obtaining one or more picture data itemsshowing images of the one or more appliances; storing the one or morepicture data items obtained in the obtaining of the one or more picturedata items; selecting, based on a specified control mode, a picture dataitem from among selection candidates including the one or more picturedata items stored in the storing; generating a presentation data item tobe presented to a user to indicate a control information item for eachof the one or more appliances which are included in the picture dataitem selected in the selecting, and superimposing the presentation dataitem at a position around the appliance in the picture data item; anddisplaying the picture data item with the presentation data itemsuperimposed thereon.

Although the control information display apparatus or the informationdisplay apparatus according to one or more aspects of the presentinvention has been described above based on the embodiments, the presentinvention is not limited to the embodiments. The present invention has ascope which covers and encompasses not only the one or more aspects butalso various modifications conceivable by a person skilled in the artand added thereto and embodiments obtainable by combining structuralelements of different embodiments without materially departing from theprinciples and spirit of the present invention.

INDUSTRIAL APPLICABILITY

The energy control apparatus, energy control method, and energy controlsystem in the above described embodiments reduce energy consumption bypresenting a user with information related to all appliances (includingappliances without any communication function) in his or her home, andcontrolling appliances with a communication function.

REFERENCE SIGNS LIST

-   -   100 Management unit    -   101 Appliance observing unit    -   102 Camera unit    -   102 x Obtaining unit    -   102 y, 1504 Image data storing unit    -   102 z Selecting unit    -   103 Appliance information storing unit    -   104 Appliance identifying unit    -   105 Input unit    -   106, 1806, 2206, 2606 Demand plan generating unit    -   107, 1807, 2207, 2607 Smart information superimposing unit    -   107 x Image processing unit    -   108, 3008 Output unit    -   109, 1809, 2609 Plan executing unit    -   110, 2610, 3010 Appliance control unit    -   151-15 n Energy consuming unit    -   301 Presentation need determining unit    -   302, 2302 Presentation data generating unit    -   303 Superimposing unit    -   1401 Home management unit    -   1402 Mobile terminal management unit    -   1403 Server management unit    -   1411, 1412, 1413 Communication unit    -   1505 Switching unit    -   1901 Demand plan selecting unit    -   2630, 3030 Appliance content managing unit    -   2701 Appliance content storing unit    -   2702 Appliance content obtaining unit    -   2703 Appliance content selecting unit    -   3105 Synchronous control unit

1-6. (canceled)
 7. An apparatus comprising: a memory which, inoperation, stores first data to identify an appliance and supplementaldata; and a processor which, in operation: obtains photographic dataincluding an appliance image; identifies the appliance image as theappliance based on the first data, the first data including feature dataof the appliance image; and displays the photographic data with thesupplemental data, wherein the supplemental data is displayed in avicinity of the appliance image.
 8. The apparatus according to claim 7,wherein the first data is three-dimensional feature data of anappliance.
 9. The apparatus according to claim 7, wherein thesupplemental data is one of text data, image data, audio data, oranimation data.
 10. The apparatus according to claim 7, wherein thesupplemental data indicates operation states of the appliance.
 11. Theapparatus according to claim 7, wherein the processor obtains thesupplemental data through a communication network.
 12. The apparatusaccording to claim 7, wherein the processor generates the supplementaldata.
 13. The apparatus according to claim 7, wherein the displayedsupplemental data is associated with the appliance.
 14. The apparatusaccording to claim 7, wherein the displayed supplemental data isassociated with the first data.
 15. A method for preparing a programexecutable by an apparatus, the program causing a memory of theapparatus to store first data to identify an appliance and supplementaldata; and the program causing a processor of the apparatus to: obtainphotographic data including an appliance image; identify the applianceimage as the appliance based on the first data, the first data includingfeature data of the appliance image; and display the photographic datawith the supplemental data, wherein the supplemental data is displayedin a vicinity of the appliance image.
 16. The method according to claim15, wherein the first data is three-dimensional feature data of anappliance.
 17. The method according to claim 15, wherein thesupplemental data is one of text data, image data, audio data, oranimation data.
 18. The method according to claim 15, wherein thesupplemental data indicates operation states of the appliance.
 19. Themethod according to claim 15, wherein the processor obtains thesupplemental data through a communication network.
 20. The methodaccording to claim 15, wherein the processor generates the supplementaldata.
 21. The method according to claim 15, wherein the displayedsupplemental data is associated with the appliance.
 22. The methodaccording to claim 15, wherein the displayed supplemental data isassociated with the first data.